0 votes
by (720 points)

I've been playing (fumbling) around with how to import data (typically stored in csvs) into twine. I have only found ways of doing this that involve the csvs to be stored online (rather than a local subfolder), which isn't ideal for me.

Then I had this idea to merely paste a csvs data into a passage and make a macro to take that passage and turn it into an object. 

<<widget "loadup" >>
<<silently>>

  <<set _db_name = $args[0] >>
  <<set _db_array = Story.get(_db_name).text.split("\n") >>
  <<set _db_object = {} >>
  <<set _db_properties = _db_array[0].split("\t") >>

  <<for _e = 1; _e < _db_array.length; _e++ >>
	<<set _myRow = _db_array[_e].split("\t") >>
	<<set _myKey = _myRow[0] >>

	<h2>_myKey</h2>

	<<set _db_object[_myKey] = {} >>

	<<for _f = 0; _f < _db_properties.length; _f++ >>
	  <<set _id = _db_properties[_f] >>
	  <<set _data = _myRow[_f] >>
	  
	  /* numbers, booleans and suchlike */
	  
	  <<if _data=="" >>
	  <<elseif !isNaN(_data) >><<set _data = parseInt(_data) >>
	  <<elseif _data == "true" >><<set _data = true >>
	  <<elseif _data == "false" >><<set _data = false >>
	  <<elseif _data == "null" >><<set _data = null >>
	  <<else>>
	  <</if>>

	  <<set _db_object[_myKey][_id] = _data >>

	  ''_id'' _data <br>
	<</for>>

  <</for>>
  
  <<set State.variables[_db_name] = JSON.parse(JSON.stringify(_db_object)) >>

<</silently>>
<</widget>>

So <<loadup weapons>> would take the passage "weapons", which is CSV (actually, tab-delimited) data, and turn it into an object.

This way I have the convenience of reading and editing the data as a csv, and then it's just one step to get it into the game. 

I'm satisfied with the result, seems to do what I want it to do without any big problems. However, whenever I do something like this and show it, invariably someone will say "why didn't you just do ____" and reveal that, through some simple method that I wasn't aware of, I could have accomplished the same thing or better with less work. 

So if you have any input on how I could do this better, that's what I'm looking for.  

1 Answer

+1 vote
by (44.7k points)

Well, the answer to whether there is a better way to store that data depends on two things: the data itself and how you're using it.

The data itself may lend itself to some system that will let you automatically generate some or all of the data, depending on what that data is.  If you can automatically generate any of that data, then you're probably better off doing that as that data is needed, since storing too much data in the game's history can slow down your game during passage transitions, saves, and loads.

A more important factor though, is how you're using that data.  If much of the data will never change throughout the game, you can reduce the history data even more by reading those values as needed from the SugarCube "setup" object variable, instead of using a story variable.  If, for example, you have a list of items that will never change during the game, you could create a "setup.items" array or generic object to store the unchanging information of those items.  You would load them up initially in either your JavaScript section or your StoryInit passage, and then you could just refer to "setup.items" to get that data.  Information on the setup object doesn't get stored in the history or saved with the game, so it should help prevent slowdown, but the fact that it isn't saved is why you can't change that data as the game is played.

Another thing I might suggest would be, instead of storing the data in a passage, save it to a JavaScript file, which you could then load using the SugarCube importScripts() function (which works with both online and local files).  Here's the code I use to do that in my JavaScript section in a few of my projects:

if (window.hasOwnProperty("storyFormat")) {
	// Change this to the path where the your HTML file is
	// located if you want to run this from inside Twine.
	setup.Path = "C:/Games/MyGame/";  // Running inside Twine application
} else {
	setup.Path = "";  // Running in a browser
}
setup.SoundPath = setup.Path + "sounds/";
setup.ImagePath = setup.Path + "images/";

setup.JSLoaded = false;
importScripts(setup.Path + "jquery-ui.js")
	.then(function() {
		setup.JSLoaded = true;
	}).catch(function(error) {
		alert("Error: Could not find file 'jquery-ui.js'.");
	}
);

You'll need to change "C:/Games/MyGame/" to your game's path so that this code will work from within Twine (Note: you may want to remove that from release versions if you don't want people to possibly know your directory path) and change "jquery-ui.js" to the name of your JavaScript file.  The above code assumes that the JavaScript file is in the same directory as the HTML file when this is used outside of Twine.  Also note that the importScripts() function is asynchronous, which means that any code after that function will be executed immediately as normal, however that JavaScript file won't actually be loaded until later on.  The .then() part is triggered to let you know when it's done importing the script(s), or if the file can't be loaded then the .catch() part will be triggered instead so you can show an error message or whatever you want there (see the following link for more information on using Promise objects like that one).

Hope that helps!  :-)

by (720 points)

That was very thorough, thank you. It does inded help. My game is already noticeably slower than it used to be during saves and loads and certain passage transitions, so the setup object will no doubt be essential when I start doing something more complicated. 

I have some follow up questions. I hope these aren't too verbose

Updating existing data: One of the things I'd been thinking about was was when to have the game check to see if new data was added to the 'Items' list (ie new items added, properties of existing items changed) so the game is using the most recent version of the info. But it seems to me that with setup, whenever you open the html file in the browser window and it shows the starting passage, it will load the data in, so the player will have the most recent Item data when they proceed to load their existing saved game. Is my understanding here correct? 

Game accessing data before it is loaded: I remember now that one of the things I was troubled by in a previous attempt was that when I had files loading in from online, the data would sometimes not be available by the you reached a point in the game when it was needed, so the passage would be filled with error messages initially until things were loaded. I assume here that with Promise I could have the game prevent the player from doing anything until the data is fully loaded (a simple dialog pop up with a loading indicator that only goes away once everything's ready). Is this how you would do it? 

Loading multiple files: I'm not great with jquery, would this be the correct way to have it load multiple files:

setup.JSLoaded = false;
importScripts(
	setup.Path + "items.js",
	setup.Path + "places.js",
	setup.Path + "characters.json"
)
	.then(function() {
		setup.JSLoaded = true;
	}).catch(function(error) {
		alert("Error: Could not find file 'jquery-ui.js'.");
	}
);

I appreciate your help. 

by (159k points)

Updating existing data: ... Is my understanding here correct? 

Yes.

The HTML file's JavaScript code is reinitialised whenever the HTML file is opened, the web-browser does a page refresh, or when the end-user uses the 'Restart' button. So engine features like the setup object are also re-initialised.

If the end-user views a later version of the HTML that contains changes to the definition/initialisation of the setup object then that object will now contain thoses changes. The same is true for any other JavaScript changes you have made as long as those changes aren't being persisted / reloaded via the 'saves'.

Game accessing data before it is loaded: ... Is this how you would do it? 

Yes, using a Promise to handle the timing of accessing background loaded data is common. 

by (44.7k points)

As Greyelf noted, the answers to questions #1 and #2 are both yes.

Regarding question #3, as I noted above, importScripts() is not a jQuery function, it's SugarCube.  I just had it loading jQuery UI in my example.  If you look at the documentation for importScripts() you can see that your way is correct for loading multiple JavaScript files concurrently (at the same time), though you'll want to change the error message you display in the .catch() to reflect the different files.

Also, just to be clear, you don't have to merely set a variable in the .then() function, that was just my simple example.  You could have that trigger some other initialization code which was waiting for those scripts to be loaded instead if you want to.

Have fun!  ;-)

by (720 points)

Thank you HiEv and greyelf. This has been very helpful! 

I need just a little bit more handholding, I'm afraid: there's some issue with the timing of the loading that I can't quite figure out. The variable in my test.js fully loads before the page even begins to display, I know this because the alert I've made to test it reports the variable while the loading spinner is still displaying. But I have <<= setup.testvar >> in PassageHeader and this shows nothing for the first passage displayed, my 'horse' only appears on all subsequent passages visited.

// in imported script file:

var loadedvar = "horse";


// in story script area:

if (setup.JSLoaded) {
	AfterLoad();
}
function AfterLoad() { 
    setup.testvar = loadedvar;
    alert("test: " + loadedvar + " is " + setup.testvar)
}

How do I get around this? Is there a way to suspend the passage loading before importScripts is called and then have it resume in the AfterLoad() function perhaps? Or maybe a way to have AfterLoad() command Twine refresh the passage? (I don't even know if what I'm saying here is conveying what I mean.)

by (44.7k points)

You're adding unnecessary steps which won't work in the order you think they will.  This is both simpler and should work:

importScripts(
	setup.Path + "items.js",
	setup.Path + "places.js",
	setup.Path + "characters.json"
)
	.then(function() {
		setup.testvar = loadedvar;
		alert("test: " + loadedvar + " is " + setup.testvar);
	}).catch(function(error) {
		alert("Error: Could not find JavaScript files.");
	}
);

The problem with your code was that the "if (setup.JSLoaded) {" line was getting checked while the script was still loading, thus setup.JSLoaded was false, thus AfterLoad() never got called.  The code above just simplifies that by putting the AfterLoad() code inside the function inside the .then() method, which gets triggered once the script has loaded.

Like I said above, you don't have to set the setup.JSLoaded variable within the .then() function, that was just an example.

Is there a way to suspend the passage loading before importScripts is called and then have it resume in the AfterLoad() function perhaps?

If you look at the second chunk of sample code for the importStyles() function it shows how you can lock the load screen until a style is loaded, and then unlock it.  Just use that with importScripts() instead.

Hope that helps!  :-)

by (720 points)

Thank you for your continued assistance HiEv. I see what you mean about the order they execute in. Only, even with the screen lock promise, I still get the same result. Let me show you what I mean.

// The twine file has one passage that looks like this:

[[Revisit this page.|start]]

If the parethesis are filled, success has been attained:
(<<= setup.IMPORTEDVARIABLE >>)

// The twine file has  this as the script:

var lsLockId = LoadScreen.lock();

importScripts("ImportTest.js")
.then(function () {
alert("It has been loaded. Proof: \""+setup.IMPORTEDVARIABLE+"\"");
LoadScreen.unlock(lsLockId);
})
.catch(function (err) {
alert("Error: Could not find JavaScript files.")
console.log(err);
});

// ImportTest.js has this in it:

SugarCube.setup.IMPORTEDVARIABLE = "I am your content, sir.";

Now, the intended result is that the screen will lock until the js is loaded, whereupon it will display the passage with the IMPORTEDVARIABLE content in its place. However, upon unlock the passage's content has been decided before the page was even locked, as the parenthesis are empty when it completes the unlock. You have to press on the 'Revisit' button to actually get the variable to display.

Having the the player have to re-visit the current passage after loading the game in the browser is going to result in all kinds of problems (changes in variables that were supposed to happen once will happen twice, etc). Is there a workaround? 

(I did some reading and experimenting on/with Engine which I won't go into because my attempts weren't interesting or successful.)

...