https://svtter.cn/en/tags/dataset/Recent content in Dataset on Svtter's BlogHugo -- gohugo.ioenMon, 24 Feb 2025 14:34:56 +0800https://svtter.cn/en/p/where-to-put-your-data-folder/Mon, 24 Feb 2025 14:34:56 +0800https://svtter.cn/en/p/where-to-put-your-data-folder/<p>When training models, we should place data and code in the same location as much as possible.</p> <p>Keeping them in the same location helps avoid path-related issues, such as needing to specify absolute paths for the data.</p> <p>For example, if I set the path to <code>./data/</code>, I only need to place the data in the <code>./data</code> directory.</p> <p>I can do this by:</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">ln -s <span class="k">$(</span>source-path-of-dataset<span class="k">)</span> ./data </span></span></code></pre></td></tr></table> </div> </div><p>To link data from other locations.</p> <p>If on the same host, git can automatically synchronize these links.</p> <p>However, if on different hosts, you need to manage them yourself.</p>https://svtter.cn/en/p/browsing-and-storing-image-datasets/Sun, 12 Jan 2025 18:31:12 +0800https://svtter.cn/en/p/browsing-and-storing-image-datasets/<p>Browsing datasets can be quite troublesome, especially when the dataset is large.</p> <p>npy (numpy array) and h5 files are two common data storage formats.<br> The drawback of h5 files is that they are prone to data corruption. I have encountered issues multiple times where h5 files could not be opened.<br> npy files have clear advantages in terms of read speed and file transfer. However, they are loaded entirely into memory at once, which can easily cause memory overflow if the server is not powerful enough.</p> <p>Common image datasets typically separate labels and images, such as COCO. This allows you to use a file browser to view images and quickly observe their characteristics. However, in most cases, we don&rsquo;t view images on a local computer but rather work with datasets on a server.</p> <p>In 2024, when working with PyTorch, I find it more convenient to directly plot images using matplotlib. Matplotlib is generally used to display a single image, but using subplots allows you to display multiple images simultaneously. If OpenCV is used, you can overlay some label values onto the images. However, there is a drawback: if you are working on a remote server, transferring generated images can consume significant bandwidth.<br> Ultimately, the choice of method depends on your own judgment!</p>