RCT Discussion / Download ALL nedesigns RCT2 objects with this Python code

  • DeathKontrol%s's Photo

    Hey everyone so I'm kind of a completionist and I wanted to have all custom RCT2 objects that are uploaded to this website but there's no "download all" button which is really annoying, you have to download them all one by one. So I got together with my friend who's a *little* more fluent with Python than I and we wrote the following code which goes through every page and downloads every downloadable object to a folder. The Python code is written to the text file I attached, just download it, OPEN IT AND READ THE COMMENTS and put in your info, then run it. It will take a while!!! I suggest disabling your computer's sleep timer and leaving it on overnight. It will download something between 10,000 and 12,000 files, totaling somewhere in the range of 500MB - 1GB (but it has to load the web page and locate the file every time).

     

    Also I strongly recommend using OpenRCT2 for this!! It seems that some of the objects can't be loaded in vanilla RCT2 (even with expansions) and may cause the launcher to crash.

     

    For those out there that aren't familiar with Python, you will need to download it here: https://www.python.org/downloads/. Then I recommend getting an editor, I like this one: https://pyzo.org. You will need to install the packages mechanicalsoup, requests, and re (use the command "pip install re" etc in the command line within the editor). Then copy-paste the text to the editor, make the necessary adjustments, save it as something with a .py extension and run it, and that's it! If it doesn't work, you can try this: https://www.pythoncentral.io/add-python-to-path-python-is-not-recognized-as-an-internal-or-external-command/. If you encounter any further problems please try to troubleshoot them on your own, I don't want to write a whole getting-started-with-python tutorial here.

     

    Hope that helps! Enjoy your boatload of new RCT2 objects!

    Attached Files

  • G Force%s's Photo
    Implementing something like this into the site would be useful for sure. Maybe on a bit smaller scale and perhaps taking advantage of the tag system.

    Regardless thanks for the contribution, certianly can be useful to those few members looking for the most complete of personal object databases.
  • SSSammy%s's Photo

    This is a great idea and thank you for sharing the code. one problem however (no fault of your own) is that there are several duplicate .dat names. i'm unsure if its even possible to work around that

  • posix%s's Photo

    Even if the names are identical, the contents aren't. They're likely neglectable though. Just, the system creates MD5 hashes that work like fingerprints for all objects, and creates new entries for what it doesn't have yet, even if the object name is the same.

  • nicman%s's Photo

    ^

  • DeathKontrol%s's Photo
    Hm thanks for pointing this out SSSammy, one work around would be for each item, before downloading check if the name is already used then if it has make a new object with a "2" at the end or something. But I'm not sure if the name of the .dat file is important, if it is then there's no workaround for that. Also I can point out that not every object is downloadable. I think I found that there are about 14,000 objects total, but at the end I had downloaded only a bit over 10,000.
     
    If either of these things can be fixed, I may try to update the code. This shouldn't be too hard. I might want to run it again myself, there were only 222 pages when I first did it :D
  • wayniepainie%s's Photo

    The RCT objects has 193 pages with a total of 12,338 objects. The other pages list the original dat's that came with the game and expansion packs. Hope this helps.

  • MrTycoonCoaster%s's Photo
    cool I will try
  • OPStellar%s's Photo

    I'm trying to reconfigure this file for Mac (for use with OpenRCT2). Thankfully, when I initially ran the script, it failed immediately because it couldn't find 'mechanicalsoup.' On second glance, I realized everything was arranged for Windows only.

    Here's the GitHub repository for MechanicalSoup.

     

    I've gotten mechanicalsoup, now, but it still doesn't recognize it as a module in Pyzo, but it works in IDLE.

     

    Also, I upgraded it a little bit by adding a feature to eliminate duplicates by preserving only the most recently uploaded downloadable version of an object under each filename. Will post soon once I finish it.

  • OPStellar%s's Photo

    All finished! I have ported the code for use in macOS (Mojave, version 10.14.6) and run it successfully using IDLE!

     

    Some important things to note:

    • Pyzo won't work since it doesn't support virtual environments ('venv'), so it can't detect MechanicalSoup.

     

    • Executing the code still requires that you have the MechanicalSoup module installed. See the installation instructions on its GitHub repository if you're unsure how to do this. (Both Python3 and MechanicalSoup can be installed from Terminal or Command Line using HomeBrew.)

     

    • I estimate a download speed of roughly one minute per page, so don't sit around waiting for it to finish! This process will demand a lot from your CPU (I've got an Intel Core i9 with 32GB of RAM 2.4 GHz, and it still ran hot), so don't attempt any of this if you're not confident in its stability.

     

    • Changing the '.txt' to '.py' is no longer necessary– unless you want to see the coloration and dissect the code. Don't change any of the coding if you don't know what you're doing! (I also don't recommend running this from Windows unless you're somewhat familiar with Python.)

     

    If you want to attempt this but don't have Python, follow this procedure to obtain and use it for this task:
    Step 1. Install Python (v3 or later) from Python's Downloads directory. IDLE automatically installs with Python.

    Step 2. Launch IDLE, then in the menu bar, select: File > Open

    Step 3. Navigate to OpenRCT2's 'object' folder* and locate rct2objscrapermac.txt** (see attached file).

    Step 4. In the menu bar, select: Run > Run Module

     

    *You should already know where to find it if you're willing to download ~355 pages of downloadable objects.

     

    **You should save this file either in the 'object' folder, or in a newly created folder within the 'object' folder. The objects will save to the same location as this document is located when you run it, and any existing objects will be overwritten if they share the same name as one downloaded through this process.

     

    THANK YOU FOR MAKING THIS PROCESS SO MUCH FASTER FOR US!! I couldn't have done it without your help (or your friend's contributions, or the help team's assistance from the Python Discord server)!

    If this ever gets outdated, reach out to me however you can!

    Attached Files

  • DeathKontrol%s's Photo

    Wow, glad this is still helping people almost 3 years later! Thanks OPStellar for your troubleshooting notes for anyone else who has having difficulty with it. I mentioned that you need to "use the command 'pip install re' etc" and by "etc" I meant that you also need to run "pip install requests" and "pip install mechanicalsoup," that should be the easiest way to install those packages. If you're using Anaconda, then I suppose you would want to try running those 3 commands but with "conda" instead of "pip." Also, looks like you had to change the directory based on where Mac installs RCT, but you got that handled.

     

    I haven't played RCT in a while nor used mechanicalsoup so I'm grateful you're willing to help whatever other crazy people out there want to download every single RCT mod ever made since I probably wouldn't be able to figure out what I originally wrote or how it was supposed to work.

  • OPStellar%s's Photo

    I mentioned that you need to "use the command 'pip install re' etc" and by "etc" I meant that you also need to run "pip install requests" and "pip install mechanicalsoup," that should be the easiest way to install those packages. If you're using Anaconda, then I suppose you would want to try running those 3 commands but with "conda" instead of "pip."
    Thanks for clarifying that! I am a nut for plugins, I get as many asI can so that it's conveniently on my system when one is required. That said, 're' and 'requests' was probably already on my system.

     

    I'm glad you dipped back in to view the achievement! Again, thank you for making this possible! My next project will be porting the Game Utilities I've gathered… eh, maybe after I finish this semester in graduate school. XD

  • OPStellar%s's Photo

    The RCT objects has 193 pages with a total of 12,338 objects. The other pages list the original dat's that came with the game and expansion packs. Hope this helps.

    I have now completed the download. 917.9 MB, 13,304 items (items #2434-27696; several get skipped in the middle, clearly), 355 pages (432 are displayed, but the first 33 pages are native objects, as are all of the objects past page 388), 3:21-8:36PM EST (about 5 hours and 15 minutes). Largest object is 5.3 MB, smallest is 214 bytes.

  • posix%s's Photo

    Why would you go through the trouble to do this? What is the goal of downloading them all?

  • OPStellar%s's Photo

     

    Why would you go through the trouble to do this? What is the goal of downloading them all?

     

    To never see this again when selecting a scenery group:

    https://pasteboard.co/3z8eMRTnWETx.png

    Unfortunately, it still happens.

     

    Also, there's apparently so many parts that they won't all show up in the small scenery selection list (it cuts off in the middle of the D's, alphabetically)! Not only that, but not all objects will even load, either! There's a cap to how many custom objects can be imported, it seems; scenery objects I know I've seen before are not showing up even when I search, namely those that started with 'zz'– so they just exceed the maximum that OpenRCT2 can handle!

     

    Also, I have a statistical update:

    534.4 MB (3,182 objects) of the downloads were 404 error pages Attached File  database of missing data.txt (41.82KB)
    downloads: 4

    Therefore, only ~383.5 MB (10,122 .DAT objects) are valid objects. I'm getting mixed results since I combined it all with existing data… so at this time, you will probably want to make sure that the final page in the range is 388 and not 432. I think I made the change already (I started the process before making the change, initially), but I'm saying it anyway just in case.

  • G Force%s's Photo

    I believe that notification is a result of the limit on scenery objects within the game, 252 small scenery, 128 large, 128 walls etc...  Not a lack of having all the .dats under that tab.  But maybe I'm mistaking here somewhere.

  • CoasterCreator9%s's Photo


    I believe that notification is a result of the limit on scenery objects within the game, 252 small scenery, 128 large, 128 walls etc...  Not a lack of having all the .dats under that tab.  But maybe I'm mistaking here somewhere.

     

    That's correct, it also happens in Vanilla+Expansions where some of the expansion pack scenery groups were made too large.