I'm uploading a set of about 73 profiles from the Solidarity Oxford dataset in one go, using a Ruby script, following information on the wiki page about Mass-uploads, and the API docs.
But I have hit a few problems, and these might be worth adding to the wiki (or at least get answers here.)
What's the maximum rate I can upload? Uploads are throttled, I only get a handful done before a limit is hit and the response comes back with this error: "429 Rate limit exceeded". Five seconds delay seems to work usually, but this will make bulk-registering large datasets infeasibly slow (some of ours would have thousands of profiles, and even 73 takes about 12 minutes).
The API docs say that "the record of a node in the index's database can be in one of five possible states: received, validated, validation_failed, posted or post_failed. The node will only be discoverable in the index when it has the status of posted". I was getting posted or validated earlier, now I seem to get received. Is there something else I need to do to get to the posted state?
There's also a slight inconvenience that the registration API call doesn't indicate if the validation passed, and a second call is required for that (I've submitted a feature request for that here).
A final thought: not that I'm not about to try registering thousands of profiles right now, but if I did, maybe it would swamp the map aggregator with pins? The reason we have so many is that we deal with membership organisations like the ICA, DotCoop, Coops UK, and lists of things like Covid 19 Mutual Aid groups, etc.
