Site Update - Testing New Image Uploads


#1

I’m currently testing a core change to image uploads for the forum which should offload images off the main server to a dedicated image server. Basically any new images uploaded from this point forwards are uploaded directly to that image server instead of directly to the main server the forum is running off. This should have some nice benefits as it adds considerably more space for storing images and should improve load times for frequently loaded images. From a usability point of view the change should be entirely seamless to you guys and this only effects new images that are uploaded with old images unaffected.

This is a relatively new implementation from the host I am using so if you notice any strange hiccups when uploading images or any drastic changes for page load times please pass along feedback. If everything performs as expected this should allow more space to start adding more historical release information for previous products which I will start doing as I continue to test. The space used by image assets along with the monthly bandwidth to serve those image assets often end up being a big chunk of costs as things grow so this should provide a more economical and scale-able solution.


pinned #2

#3

Cool, saving money is a good thing.


#4

Definitely. It is a much better option as the image server space can be scaled in size independently. Normally when you are paying for server hosting you can’t just upgrade storage space, they usually price upgrades in tiers that include mandatory bumps in cpu and memory which you may not need, or they try to move you over to a more expensive dedicated server. This way it’s just paying for the storage size you need and the bandwith you use.


#5

I will let you know when I do my next review.


#6

So far from testing it seems quite stable, just a small delay while processing images while they are transferred to the image server.


#7

When looking at how to organize a historical product announcement archive I notice performance hits from creating too many subcategories as you can get dozens and dozens of brands. It also makes for a very cluttered interface so i’ve implemented tagging instead as a way of more quickly navigating a larger category like this will be. As in the below example you’ll notice I’ve started tagging product announcements with their brand name, which appears below the title. You can click on these tags to bring up a list of all related topics for that tag, or brand, in this case.


#8

At the bottom of the page you’ll see a new section, Product Announcement Archives. I’ve added some posts in there for testing the new image server space. When you have a chance visit some of the topics in there that have a lot of images (Playhouse and DAM release are good for this) and let me know what the image load speeds are like when you load the topic. Some of the more image intensive ones will load up to 80 image previews at a time so it won’t be unusual for a few images to load slower as you scroll up and down the post.

For newer posts there is a delay before any new uploads get cached so they may load slightly slower. From my testing to so far performance seems reasonable. For comparison this topic has the images stored natively on the main server which uses faster SSD drives, so that should load a bit faster than content on the separate image server. Once the newly uploaded images get cached, they should be served from a variety of cache servers in different geographical locations which should hopefully give decent load times for people in different areas.

For now the section is read only while I continue to test things.


#9

Added tag navigation to the menu bars, this allows for another navigation option if you just want to see releases from a particular brand.


#10

Site will continue to be in read only mode for a while as I’m testing some things which need the database locked down.


#11

Images have all been migrated over to the image server. Unfortunately we can’t have a mix of locally stored images on the server and remotely stored images as it breaks the lightbox feature, which is what gives you the full size image pop up with carousel to scroll through photos.

Still working through and fixing some the posts where images are still not lightboxed properly. This process is quite cpu/memory intensive as it involves creating and resizing photos in various sizes to optimize for previews, full size, mobile, etc, so some areas of the site will remain locked down while that happens. This is mostly linked to threads with a large number of images in high resolution, like @chpo reviews or some of the larger announcement threads where there might be 50-80 manufacturer photos.

The original image size and resolution have an impact on how quickly images load and how quickly the lightbox/full size photos are available for viewing once upload. The way image uploads work is that anything larger than the thumbnail view (the normal size you see in a thread) maximum resolution gets resized in both size and resolution in multiple versions, so there may be 5-6 images created from one image. These also have to transfer over to the image server at the same time. With larger threads that have many images this gets resource intensive both in processing time and space used.

To tweak performance on this I’ve set new limits on uploads. The maximum upload size is 5 megapixel (minimum software allows), 5 megabyte max original upload. The forum software will then resize the image to hit a maximum file size of 500 kilobytes. The 500kb change is the big one as before it was allowing images up to 5 times that size. Anything file uipload over 5mb will prompt an error.

In practice this change should result in photos sizes pretty similar to the typical manufacturer promo shots and as long as your individual photos are under 5mb the process should be transparent to you. Some sections remain locked down while I fix some of the threads where the lightbox isn’t working.

Here is a before/after example of the changes, original image upload compared to the new resize:



#12

Server for the site has been migrated to the same datacenter that hosts the image server space. Hopefully this should improve performance a bit with the two on the same internal network.


#13

Still tweaking some things with image uploads. One of the current challenges is how the forum software resizes large images that are both large in mb and resolution. It takes an approach of prioritizing the downsizing of the resolution rather than compressing the image aggressively. With very high resolution images (like you would get with the default multi-megapixel nature of cameras/smartphones) that are 2-5mb or greater they won’t always compress the file size to the 500kb target resulting in an error message for uploads. I’ve raised the limit a bit higher to 800kb to account for this so if you get an error when uploading a large file try reducing the resolution to something more reasonable like 1920x1xxx first. I’ve also found that on occasion an upload that is sufficiently small will fail with an error message. When this has happened opening the file in an image editing program or a resizing program and compressing it slightly (ie 99% quality) will usually reformat the image properly and allow it to be uploaded.

Image load times are are fairing a bit better since migrating the server to the same datacenter as where images are hosted, the biggest slowdown compared to when we used to host the files locally on ssd storage is how long it takes for the lightbox preview gallery (full size pop up view) to process. For a larger thread with a larger number of photos it can take a few minutes to process and upload all the various sizes. So far the performance trade off seems acceptable as the equivalent ssd storage space on local storage would be 4-5x the overall monthly cost.


#14

Still addressing some performance issues related to the move over to remote image storage. Lightbox processing time is still problematic, so if you don’t see the pop up image gallery available for new topic uploads it is being processed in the background and will automatic enable itself once that processing is done. In really image intensive threads with many large high res photos that processing time can sometimes hit 15+ minutes. I’ve made some temporary improvements and once some requested code changes are done it should be greatly improved based on the manual testing I’ve been doing in the new release/product archive sections.

There are still some periodic issues with images loading in threads with high volume of images, but usually by the time you scroll up and down a few times it resolves itself.

If you receive upload errors when uploading a batch of files, try reducing to a maximum of 10 at a time. There are several limits at play here, how many total files can be uploaded at once without the server triggering a rate limit (antispam) response along with the total size of the files being uploaded. If you consistently get an upload error stating a file is too large, use an image processing program to compress the quality a bit. With high resolution files the upload feature can only reduce the resolution in fractions to reduce the size of an image, and in some cases with un-optimized photos (ie raw from a camera/phone without much compression) it can’t hit the target maximum image size by resolution reduction alone.


#16

You’ll notice a change now with how images load. Previously the forum handled multiple images posts by loading chunks of 20 images at a time. With the large number of images in threads combined with the images not loading in sequence this would make for an uneven viewing experience. Images now lazy load which makes them load as you scroll down and they come into your viewing area. So far from testing this seems to be a nice improvement for the image heavy posts where there may be 100+ images in a post.


#17

I like this one better now.