Topic: Land ahoy! Seamless transition in maps? (Read 8755 times)

Holy Diver · « **Reply #30 on:** January 04, 2022, 05:27:51 AM »

much better

I had some luck getting those load times back to normal. There's no information on this online that I could find, so this would be a valuable resource if I knew how to get it out to programmers who would stand to benefit from it.

Pretty much how Microsoft insists on getting shortcut targets turns out to be extremely slow for no good reason.

There was one resource (links in code below) that thought they'd figured it out, but their code wasn't good for real shortcut files... maybe it was back in the day, but I doubt it. Fortunately their posted code cited a source (link in the code below) and looking at it gave me an idea to see if the IDLIST section was actually the target, and sure enough it's possible to just use SHGetPathFromIDList and be done with it.

One weird note (in the code) is the supposed "points to a file or directory" flag/bit did not hold true for all of my shortcuts that were BMP to TXR shortcuts. This makes no sense. These are files, just image files, but they should have had this flag set. So it shouldn't be trusted or means something else I guess. Note these shortcuts are made with IShellLink (by x2mdl.dll) and I can't see a way to avoid that, however reading them is better done this way.

For comparison my KFII project was taking over 3 seconds to load, but with this change it's down to around 0.8 to 1 second (varies) which is within margin of error of what it takes without any art/shortcuts involved. So it's safe to say that IShellLink is responsible for all of that overhead (note, that using shortcuts as time-critical timestamps is maybe an unorthodox use case, however time always matters in any application to some extent.)

Code: [Select]

struct som_art_lnk //DEBUGGER
{
	char magic[4];
	char guid[16];
	DWORD idlist:1,path:1;
	char _unspecified[52];
	///idlist data goes here if present
	WORD idlist_sz;
};
static bool som_art_shortcut(wchar_t lnk[MAX_PATH])
{
	FILE *f = _wfopen(lnk,L"rb"); if(!f) return false;

	enum{ sizeof_buf=4096 }; //union
	union
	{
		char buf[sizeof_buf];
		wchar_t wbuf[sizeof_buf/2];
	};
	int sz = fread(buf,1,4096,f); fclose(f);

	char *eof = buf+sz;

	//source (code is pretty iffy)
	//https://www.codeproject.com/Articles/24001/Workaround-for-IShellLink-GetPath
	//sites this source (its code doesn't get very far)
	//https://ithreats.files.wordpress.com/2009/05/lnk_the_windows_shortcut_file_format.pdf
	
	auto &l = *(som_art_lnk*)buf;
	//for some reason BMP->TXR links have path set to 0???
	//if(sz<sizeof(l)||!l.path) return false;
	if(sz<sizeof(l)) return false;

	int a = l.idlist?l.idlist_sz:-6;
	if(buf+a+78>eof) return false;

	if(l.idlist) //TESTING
	{
		//don't know why the codeproject source didn't just do this!
		return SHGetPathFromIDList((PCIDLIST_ABSOLUTE)(buf+78),lnk);		
	}
	else assert(0); return false;

////NOTE: ALL OF THE BELOW CODE IS DISABLED////

	/*this reproduces the codeproject site's code
	//what I got was a string to "C:\Users\" even
	//though my path was ANSI
	DWORD &b = (DWORD&)buf[78+4+a];
	if((void*)&b>eof-4) return false;
	DWORD &c = (DWORD&)buf[78+a+b];
	if((void*)&c>eof-4) return false;

	char *str = buf+78+a+b+c;
	char *e = str; while(e<eof&&*e) e++;

	if(str==e||e==eof||e-str>MAX_PATH-1) return false;

	for(int i=0;str<=e;i++) wbuf[i] = *str++;

	if(!GetLongPathNameW(wbuf,lnk,MAX_PATH)) wcscpy(lnk,wbuf);

	return true;*/
}

EDITED: I should try to make a patch with this fix. I have to try to disable all of the effects from any WIP code first. I'm relieved this worked out. And surprised I didn't noticed the load times being that much more.

Holy Diver · « **Reply #31 on:** January 05, 2022, 02:29:44 AM »

I'm published a patch for this. Funny story, I just checked and now my test map does load instantly. It suggests most of the pain was coming from this shortcuts problem. How funny. Perfect timing... but on the plus side it brought my attention to this problem.

Now I have to figure out a new test framework!!

Holy Diver · « **Reply #32 on:** January 10, 2022, 09:52:37 AM »

hard problems

Something that's been bothering me is transitioning lighting is not going to be so easy no matter what. KFII and KFIII get around this by keeping the same lighting throughout, I believe.

This might be a time to think about the "Dark Slayer" system, since it would be able to blend the lights. Although, I think that it might not be ideal for lights in the distance to receive the same amount of influence from the outgoing map. In that case it wouldn't help to treat the level geometry like "objects" in terms of lighting. On the other hand probably a SOM game's draw distance won't be that great, but it can be pretty great.

So the regular (vertex color) system might have a leg up here. To do a distanced weighted lighting on the GPU would mean doubling the amount of light sources, which might not make sense. This makes me to think that even under a Dark Slayer system it might be best to represent the direction-based lighting with vertex colors. Except the problem with that is it would not work well for an alternative "per-pixel" model (i.e. doing away with vertex interpolation.)

Another chief problem is the map exit point isn't known until the event fires. That means there's no way to blend the lighting from the incoming map prior to the transfer. That will mean there will always be an asymmetry depending on if coming or going.

Simply not blending is not an option either. That would result in a visual glitch.

It suggests that some kind of more advanced stitching system is in order.

One strategy I have in mind is to reverse engineer MapComp's light model so that the player can regenerate the level geometry lighting in real-time in order to lend a dynamism for some effects. This is a very possible first attempt to solve this problem. It's less dramatic than throwing myself into a Dark Slayer project, and more robust.

For my KFII project light blending is probably not needed, and unless someone comes along and puts this new system to good use I expect this is something I'll neglect to implement for some time after everything else.

One more baroque solution that many games might implement is to build a liminal tunnel with the properties of both maps/ends. However one of the cooler things about SOM is it never cheats. It does everything generically. No smoke and mirrors. So I don't see this as a viable strategy, especially because it would entail a lot of complexity for a limited solution.

bitmap system to the rescue

A left field idea I have just now is to work on the light painting system that ties into the bitmap extensions to be able to make sections of maps have different lighting. This is actually the best solution. I think this is why it helps to write posts about your work... ideas come when writing things down. Spelling things out.

Originally I imagined this system would be completely one set of lights or another. I wonder if there's any way to make it a soft system. Somehow a system that can blend gradually.

Even if there's not way to do that (other than perhaps doing so universally) it would still offer a manual way to just not blend the lighting. To look at it a different way, it would be the "liminal tunnel" or one way to construct one... made of light. Symmetrical. An other upside of course is such a system will open up new creative possibilities.

I still have to add the bitmap painting functionality to SOM_MAP. In the English language pack there's been buttons and readouts for this for a while, but I haven't gotten to make it interactive. It's possible to just use any paint software to edit the bitmaps, so it's not been essential to my KFII project... which doesn't even require that because it gets its bitmaps off KFII's CD-ROM.

The bitmap system can be used for anything, not just lights. It's a way to tag tiles with information. The bitmap color can be applied to tiles as a lens too, which is what KFII does. Although it's a bit more complex since it translates into contrast and darkness.

a novel idea (this could work!)

I'm not sure yet how to assign lights to bitmap colors. Technically it would be set up in the Ex.ini file, but I intend to leverage SOM_MAP to use it as a lighting studio, since it would be too complicated (inelegant) to add a new editor when one already exists. Still I'm not sure how to make it easy to accomplish this, or how or if to include a low barrier to entry mode, as in a less flexible but more intuitive starter option. One idea I just had is to completely replace the tile palette in this mode with a color palette. This could itself be a bitmap perhaps. And maybe double-clicking the palette or pressing the button under the tile preview could open the light setup window for that color. And unchecking all boxes would delete a light. A light can be indicated by an overlay. Then it would be nice to be able to assign a color to the palette. I think maybe that can be by using the same system as assigning to a tile except when the palette is selected. The main difference being that you can't draw onto the palette since it would not make sense to.

Holy Diver · « **Reply #33 on:** January 12, 2022, 01:06:53 AM »

progress

Last minute last night I got the controls stuff straightened out... not actually limited to controls since it was important to not drop any frames either. Besides controls the main problems were two-fold, some frames were dropped on the map change, but also there was a bug from ancient SomEx code I didn't understand where the first frame after changing maps would flash what looked like garbage, so I'd just set it to ignore two frames in haste. I have to return to that since the reason it happened was triggered by there appearing to be two frames drawn without submitting them (SOM used to have these crazy runaways that would go on for hundreds of very fast frames this was aimed to minimize.)

Somehow in the course of this work I'd come across all of the relevant code, so I was familiar with everything. It's sometimes hard to believe I don't really have a grasp on any of the code I work on. I don't know if I don't have a memory for it, or if I just move from project to project, but the truth is I don't have a real knowledge base, just experience at quickly finding my bearings. Programming to me is not something you can learn to do, it's not even about ongoing learning... it's more about being so deep into something completely different everyday that nothing ever sticks, nothing is ever truly learned, everything must always be forgotten, everyday, so it's more instinct that builds up. At least that's how it is with my brain. It might be helpful to have a brain that's compatible with this kind of never repetitive work.

I finally knew something was wrong after I got the load time down to 0 milliseconds and ruled out control effects. I've got new code in place for faster duplicate texture lookup (the original code does a linear search, which is very slow) that helps to not trigger the art system's scan. And I put in an SFX cache for remembering if SFX models are TXR or MDL files. I need to do the same for SND today.

I have a simple "thread" (likely to use another core on your CPU) for loading the maps. Today I have to teach it to preload models. The resource pools need to be owned by this thread or the main thread, but mainly it's just menus that need to takeover ownership, besides the MPX loader (i.e. level change) and that's the extent of the thread synchronization logic, except for managing a jobs queue.

event options

I think I'm going to add a "Blend Fog" and "Blend Sky" options to SOM_MAP's map change event. It's a little bit different semantics than the others but I'm not sure it warrants its own section. I may make that default and make the relative mode default too.

I have a note to add rotation to the trip zone event protocol. I think it will just copy the rotation of the subject it's bound too. I hope there's room for a checkbox for this and to add a height option as well, since height will be essential for layers (you don't want your event to be tripped on the wrong layer/level.)

more thoughts

Something I'm eventually aiming for is to go ahead and jump into the level (finish loading) as soon as everything that's in the visibility radius is loaded and postpone loading the rest... or more likely just keep loading it based on distance... as opposed to waiting until things come into range... we'll see.

I haven't decided yet if this is something I want to squeeze into this project. It might be necessity if I tie the release of this work to my KFII project. I will probably put out a demo if I do. If I can't get KFII's maps to load fast enough I will have to add this extra logic to it then.

In any case, good news is it's coming along!! It's gonna be

Holy Diver · « **Reply #34 on:** January 16, 2022, 05:54:44 AM »

update

I just managed to write all of the code for background loading models (textures) sounds and special effects in one day yesterday. I didn't get to try it out though.

Today I have a lot of distractions to tend to, I may not get to work on or try out anything. I had to do laundry this morning. Yesterday I found myself looking at the PlayStation network to see if there was anything new for PSVR or cheap games. I got a game called Kingdom Come "Deliverance" I've been wanting to try for the price of a rental. I played its intro today while doing laundry. I'm impressed with its presentation's un-video-game-like maturity. I'm unsure how a game such as itself compares to a King's Field experience. I was relieved it was first-person (I couldn't remember) when it came on. Its art is very nice but also very rough around the edges. To take food off of the table in the starting room it's necessary to squat down and climb onto the table since your hands can't otherwise reach it, and that is even though there's a sit down function, which is nice, which could let me sit and eat the food like a civilized human being, however in this mode it's impossible to interact with the food items, even though they have labels applied to them that can't otherwise be accessed unless doing the squatting/climbing maneuver.

It does baffle me how games can be both so elaborate and seemingly well crafted and unpolished at the same time. Maybe like Witcher III it's a masterful way of lowering expectations out of the gate.

Anyway, I do worry it's an "open world" game, and I swore off those after Witcher III (the only open-world game I've played) but might make an exception if particular games have other intriguing possibilities. I'm uncertain yet if Kingdom Come will become a temporary distraction from my work. I think I also want to try Hideo Kojima's latest game, which is pretty open-world too, but it wasn't part of the same holiday (in January?) sale. I guess it still costs serious change to buy it.

Staying on this topic, I'm really surprised how unappetizing most games are to me today. It feels strange since games were a big part of my teenage years. It feels like this world isn't real, and it's conspiring to ensure I don't play games anymore. I just don't understand why average people find the games on offer appealing. They're all so uniformly similar too.

Edited: Oh yes, back when I first got new fiber Internet (not too long ago) I tried Sony's PlayStation Now service for one month. I didn't have much time to spend with it, and it disappointed me that many games couldn't be played via the streaming feature (what I wanted) instead requiring full download. I also didn't like that the streaming games are stuck in a slightly lower resolution, but I think it's probably something to do with compression and downsampling the streamed image to hide compression artifacts. Anyway, the only game I thought was interesting that it had on offer was called Detroit "Become Human". I know its studio and director isn't well liked by gamers, but I thought the game was impressive, the single scenario I played with it. I did feel a little like the control scheme wasn't intuitive or was clumsy, and didn't like the time pressure aspects, however, I imagine that could shake out with practice and it might be a very impressive game. I don't know if I will ever return to it, but it made an impression on me. [I definitely would return to it if I had more free time for video games... or anything for that matter.]

Holy Diver · « **Reply #35 on:** January 21, 2022, 04:48:58 PM »

Here's a preview of the new feature set and layout I'm aiming for to add more flexibility to event triggers. I kind of got carried away in giving two settings for the vertical range options. I was thinking an event instruction can be extended pretty easily, but this is the fixed starting part of the event data, which doesn't have much room for expansion.

I didn't want to toss the layout, so what I have planned is once those boxes are checked it will technically throw the event into a new hidden "protocol" which will have to be managed somehow. The other new feature is the "Align" box, which allows for some ability to rotate the trip zones. I'm not sure if it will rotate on the lateral axes or not. I limited it to the X/Y mode since rotating a circle doesn't make sense, however technically it could if the lateral axes are included, except I still feel like the circle mode is just a simplified mode.

I'm thinking more in terms of diagonal walls and inclines. Inclines would require lateral rotation, however if so I think it would be implemented more as a projection, and likewise I expect the vertical range to not be rotated. Pretty much what's needed is a way to trigger map changes over irregular spaces. The vertical part is also needed to confine events to layers.

The vertical can also be applied to the regular cone protocol. I just decided it wouldn't fit into the "trip zone" area and it makes sense to offer it with the other protocol. There's just enough data to cram all of this into the event data with a new hidden protocol to cover all three. It won't be backward compatible.

Holy Diver · « **Reply #36 on:** January 23, 2022, 02:00:41 PM »

I think I bit off more than I could chew the other day with that design, but I didn't want to redo it or scale back so I ended up thinking outside the box to make it work. (There's still a lot of work to do on it and I don't know if I will get time to do much work today.)

The problem was SOM_MAP's internal representation of the EVT data is slightly different, and doesn't include the extra unused "padding" in the same places. I also realized there's a 4th mode I forgot that's activating the event with an item, so I couldn't use the item slot to encode those checkboxes. (Luckily there's some padding at the end of the "predicate" data that's even better since it's unused data. It's just a little out of the way.)

The only thing I could come up with to make this work without making a mess of SOM_MAP was to compress the new data when the EVT file is loaded, and decompress it when it's saved or the event is edited. Compression is possible because the trip zone is actually limited to a single decimal place, and 200 meters is the maximum, so this can be encoded into 12 bits with a sign for negative values.

Usually I remove these arbitrary restrictions on decimal places where I can, but this was the only way I could make this work for now (without too much butchery) and great precision isn't really required for trip zones, etc. since they're invisible to the eye.

Off-topic: this morning I found myself solving some problems with the various mouse states starting with a new problem where the captured mouse actually goes behind the window on start up, which is a bug. I thought this was a new policy in Windows, but this morning I had an idea to test my (pretty old now) demo on itch.io to see if the bug was there too. It actually was, but I realized the problem might not be with the SetCapture API but with ClipCursor, and that was right. I don't know if I deleted some code by accident or what happened (the cursor logic is actually unbelievably complicated.)

But that was just the beginning since whenever I look at mouse cursor stuff I always discover new problems. It's ironic because the biggest problem with the mouse stuff is PC gamers keep wanting to play King's Field with a mouse! Anyway, I ended up working on a few things, for a while, which will be in the next release or patch. One was a pretty serious bug where somehow when in weak capture mode clicking in menus was registering two button presses (the second on releasing the mouse button, oddly.)

Holy Diver · « **Reply #37 on:** January 28, 2022, 08:25:30 PM »

update

Sky and fog blending is working now satisfactorily based on traveling distance. The metric I chose is the minimum of the two involved maps' default draw distance. I.e the one set in SOM_MAP and not one generated by Ex.ini extensions. Really the exiting map's distance would make more sense here, but I used the minimum to avoid weird temporal asymmetries when crossing back-and-forth. I might try to compute something better sometime. It might be nice to be in control of this (via extension) some day.

The pedals/inertial system is used to bias a dead-zone a little bit. Inside the dead zone nothing changes. It's a 2m diameter. It might not be big enough to fully avoid transitioning the sky when going backward from the landing point. But the transition is very slight and most people would walk on through instead of stopping and going backward.

When doing this kind of transition the event settings for changing the directions and offset really don't make sense and will probably cause a glitch if used. I may have them ignored. I think it would be useful to repurpose them to communicate the orientation of the blending operation. This would prevent the backward problem. (I.e. currently the orientation is assumed by the direction of movement while crossing the threshold.)

For the extensions that control the fog layers, what I ended up doing is retain the old values for the exiting sky. This allows events to drive them without introducing anything new. I'm planning to convert the "powers" (exponential) extensions to shader variables so they can be blended and changed on the fly by extensions. Currently they're hardwired into the GPU shaders, so they must be the same on all maps. This could easily take a whole afternoon, so I don't know if I have it in me to do it before this release because I'm already feeling like I'm ready to publish it so I can do something different for a change.

unfinished work

I still have to do the Japanese layouts. There's a lot of them. That will take a day probably. Otherwise all that's left is the music change on map changes. I probably mentioned it before but, unrelated/related I modified the map change to not reset the BGM if maps use the same music. This way it will feel like one large zone. This reminds me, I was wondering this morning if it's possible that the reason DirectSound can monopolize the CPU could have something to do with the fact that the BGM is streamed out of an open file (I/O) object. I just wonder if that is compatible with a 60fps app running on the same thread. It depends on if the generic FILE subsystem is smart about buffering enough data to fill the sound buffer.

I'm not sure yet if I'm going to add some more options to the music events. My goal is to confirm that music isn't faded out. If it isn't then I have to solve that problem so the exiting BGM can fade out without a jarring cutoff. Until I add some event options I'll assume the new map's BGM starts out from silence as most music already does so.

thoughts

When I say blending the fog, this includes draw distance and sky fog and the background color (which is the fog color.) How this is controlled is the map transfer event fading settings. There's 2 new options that are now the default called Blend Fog and Blend Sky in English. They're not actually in/out options but they use the same slots. You'd probably always want to apply the former. There's no real benefit to not doing so. The latter you might opt out of if the sky isn't visible. Drawing two skies at once can be costly and could cause your frame-rate to drop.

This just leaves out lighting. Like I said, in an earlier post, the level geometry doesn't do lighting in a way that would be easy to blend. Everything else does. But I'm not sure it would look very natural for the light to blend anyway. It definitely would for the ambient, but to have the directional lights seeming to reorient themselves might be disorienting. Unfortunately there's no solution for this coming in this release I'm preparing. I can't jump directly into working on it since I need to feel like I've taken a break from this line of work and it's going to be pretty big project to add a lot of features to SOM_MAP, which is how I plan to address this. Unfortunately even after I can finish that it's going to be a little complicated to work with lighting groups. Artists will just have to learn the ropes to make lighting transitions work. Of course you can also keep the same lighting scheme throughout. I think KFII does this but I'm not sure. (It has another system system for supplying ambient light levels to tiles. This couldn't be achieved by itself with this new system since even to override the ambient setting would not be the same effect. My KFII demo implements it via a set of extensions. However the new additions to SOM_MAP could be used to fill out the BMP files these extensions use. FYI it will pretty much just use the existing overlay system, but open up a new possibility to draw directly onto the bitmaps with SOM_MAP.)

Holy Diver · « **Reply #38 on:** January 31, 2022, 03:17:55 AM »

maybe tomorrow

I had a half day today and got the BGM transition working, although the kind of hack system that can convert non WAV formats causes a hiccup even though I'm pretty sure it runs on its own thread. I think the music files need to be preloaded or opened at least too, but I'm not going to work on that now.

It would be nice to have an option for how fast/slow to fade the music out, however I think there needs to be more options for fading in/out music in the music event things, and I think it will make sense to do that all as one project.

One weird thing about cutting the volume is technically silence is defined as a huge number (-10000) and I think that almost all systems will go silent long before hitting that number, so there's a risk of a lot of dead air. I went with a 3 second transition. There's not much pause. I tried some more extreme power functions to speed through the dead air, but I fell back to a simpler function and using the log-2 function that the volume controls use, even though I thought it sounded better without it. I really was surprised by that because I figured it would sound as bad as the old volume controls without log-2. I'm trying to think if the volume extensions should automatically do log-2 conversion to make them more human-friendly. I think maybe I should have used log-2 when controlling the footstep volume.

Speaking of footsteps. I did the Japanese layout work all day yesterday. I know one last thing before I can maybe publish this tomorrow is the 3D sounds cut out when doing the map transition because their 3D locations need to be translated to the new map's coordinate system. Their not gone IOW, they're just dislocated in space and so most likely inaudible. That should be simple enough. I'm not sure what other little odds and ends I've left for the end. I put little traps for myself in the code that will prevent it from working after I increase the version number. If there's not many of those I should be able to put it out tomorrow and write a blog post. I'm not sure how to feel about this. It's really pretty cool to have this stuff working, finally. I feel like I should be super excited, but it's kind of hard to get too excited if no one shows up to celebrate. It's always a cold flame for me and SOM.

Holy Diver · « **Reply #39 on:** February 26, 2022, 02:12:15 PM »

somewhat off-topic idea

Lately I've been organizing tiles for my KF2 project's second zone. I have all the tiles in place now, however, there's more of its palette swapping system in play on its map than in the first zone.

I haven't yet begun trying to untangle this part of it. I have to regenerate the map with its duplicate tiles (duplicated from the first map) substituted for the tiles on the first map, since part of this project is to unify its maps into a single tile set (in reality they were probably unified in From Software's map data but run through a compilation step--not unlike MapComp--that results in the duplication... although it's possible too they were always duplicated, in a disorganized fashion.)

But it's given me some time to think. What I think is palette swapping textures is actually a powerful system for injecting variety into environments. Especially in a visual presentation like KF's where the 3D model data is so minimal that oftentimes the only distinguishing factor is their "texture".

I believe when I begin to work on the lighting solution I've described already in this topic/thread, it will likely benefit from a new UI element that allows for choosing a "palette" for SOM_MAP's palette area. I believe at the same time I will try to invent a way to describe a palette for regular tiles with alternative textures. What working on KF2 helped me to see is such a palette system shouldn't be a map-wide swap, and also it shouldn't work like KF2's system, where textures are swapped out on the fly either. Instead it needs to simply inject the alternative textures into the MPX file, and they won't count as additional tiles toward its 1024 tile limit either.

Such a system will need to swap out the palette area in order to control how the MSM model preview appears. Selecting tiles on the map will automatically change this palette so it can highlight the selected tile.

It will be cramped, but I think there's room for a button labeled "Sets..." and a drop-down with about 20 characters for a name. Usually SOM's labels are 30 characters, except the monster attack labels are 20 characters, so 20 is probably fine for palette names. (As for designing a palette, that's likely to be a very big project in its own right. The UI will be anyway. Functionally I think it will require choosing from existing PRT files, and assigning them custom textures, which will probably be inputted with text fields. That's a little bit low-level but I don't know what else to do. It could also be a back-door system for assigning material properties to tiles with some more thought. The bitmap/lighting/etc. system will have its own set of palettes. It can't store its palettes in ts bitmaps so it will be a more manual/advanced, general purpose system.)

EDITED: I meant to add, although KF2 has inspired this, I don't intend to use it with KF2. For one, KF2's use of palette-swaps is far too disorganized/chaotic, so it wouldn't add value to use such a system with it, and (for two) it's a known quantity and its models are simple enough, and I wouldn't say its use of palettes is appropriate for the relative simplicity of its design. As an aside, in order to make MSM files more useful for palette swapping I can see it being good to break up parts of the UV map into groups even if the MSM file only uses one texture, in order to enable mixing and matching textures. That might be best suited to a metadata format that the SoftMSM system is likely to require in order to better support working out of temporary files with the new art system. (BTW, I talked about materials for MSM. I'm also thinking about letting MDO and even MDL work in place of MSM files. That would enable materials, but I might start that with this palette system since it will be easier than supporting alternative model formats in MapComp. Once that works there will be more incentive to do the harder work.)

EDITED: Another thing I've long been interested in is alternative UV maps or even full tile variants. With the former you could rotate the UV mapping to get different effects, and avoid that kaleidoscopic feel SOM has (on ceilings and floors especially) if you really want to. I don't see a way to put this to use without adding a "context menu" system to SOM_MAP, which I don't plan to do until my Exselector project is published. However I mention it, since I think it would be a good justification for that metadata format. In fact, it might be a good way of thinking abut what that format might be. It suggests that it's more of a model format than a PRT accessory format, which is a good thing I believe. When I'm designing a new version of the MAP format (version #14) to incorporate palette swaps I will have keep this in mind.

Holy Diver · « **Reply #40 on:** March 13, 2022, 09:58:26 AM »

KF2 report (not great results)

Okay, so I said I couldn't sell this without testing it with a real project, and now I can report the results are in on that.

I'm not happy about it, but with no disk access crossing maps, it still (somehow) takes a very long time to swap the memory over for the map change. It takes my project (for some reason) going both ways, 120 milliseconds to change maps, which is either 8 times longer than 1 frame, or in real terms, you can't even count it in frames, because you can't afford to sacrifice a frame (drop a frame) so the target needs to be down much lower, to like a few milliseconds tops, ideally under a millisecond.

So you can say it's 120 times too much... but that's not really what's going on here. What's going on has something to do with C++'s built in memory manager. In short, it's unfit for games really. Professional games avoid it pretty much. So what's called for now is to labor to build a new/custom memory management system. I won't lie, this probably won't make the cut in my new project demo I'm working on now...

Pretty much it was a big project to eliminate disk access, but only one leg of the job. I'm not sure how I should approach this new leg... there's a few different options. I will have to think about it, and maybe go with a mix. The most drastic option would be to hollow out the old memory call sites som_db.exe uses. I will actually probably try this first as it's the simplest to do quick and hope for the best. I might be able to find some off the shelf memory management code that I can plug into it. Another way is to target the map's memory... i.e. the actual problem case, and just try to get everything ready in advance so there's no actual memory allocation when the change over happens. I think the latter is probably a must unless a generic solution is such that it essentially accomplishes this just by being more basic than the standard C++ one... one other thing I can try is it's possible the old code som_db.exe uses is incompatible with whatever advances have come since, and maybe if I just plug it into the more recent implementation (courtesy Microsoft) it will do better, but this seems unlikely, these things haven't changed (interface wise) as far as I know since back in the day.

Anyway, long story short, it's not out of the woods yet. But it's half way there. (And probably--knock on wood--the first half will have been the harder half.)

Holy Diver · « **Reply #41 on:** April 07, 2022, 03:16:13 AM »

update

I pulled a long day yesterday working on changing sky models in the middle of the map for KF2's second zone (blue sky over praying Nola's house.) It sucked me deep into multi-thread synchronization territory, which is a very scary aspect of modern day programming. I think I managed to work out all the kinks, and it's made me have a wild urge to tackle this memory allocation optimization problem very soon (like today/tomorrow) to be able to realize perfect map transfer with any luck.

off-topic

Another problem that's arisen with skies is the blue sky doesn't make sense as an animated sky. I suppose I could slip a dummy layer into the model to make the blue part stay still, but I'm taking it as opportunity to inject a little bit of sanity into SOM's scattered/selective UV animation abilities. I think my plan is to add this kind of basic UV animation to my MM3D modeling project, so it can be edited as a normal aspect of SOM development. The old Assimp (x2mdl) code I'm using has an aiUVTransform property for materials that's not used for UV animation, but it can represent some basic parameters which otherwise would have to be communicated by more tortured means or just added to the data model which it uses. If I'm going to hack it I prefer the lightest touch possible. I do have my own rewrite of Assimp, but I'm not using it to develop x2mdl at this time. I plan to rewrite x2mdl with that Assimp rewrite at some point when I have a big chunk of free time in my schedule and feel like its time to. Adding this new information to the MDO format shouldn't be a problem because I've already developed an open-ended extension model for it that is so far used to link it to MDL animation data. I don't know if I foresee full UV animation like MDL does with regular vertices, however I think it would be useful to try to represent the kind of animations the SFX system uses, which can include movie like playback. It would be nice to be able to model these things directly in editing software.

Holy Diver · « **Reply #42 on:** April 10, 2022, 11:49:20 AM »

progress

I've been hard at work the past few days and finally I can pass between the first two maps in my KF2 project without interruption.

I just figured out an important bug around materials. It keeps a count of how many materials are in use, but this is almost a meaningless value because the material index is a sparse table as soon as a material is removed. So the material retrieval code wouldn't hand back any materials with indices higher than this counter. Really it should be checking the capacity and not the counter. The upside of this is some things like sky and SFX model instances are a lot easier to manage. I decided I had to get to the bottom of this.

Another thing I just figured out is I must've screwed up something because the job (task) that was supposed to be mipmapping/filtering textures in the background wasn't being issued at all more or less. What's surprising is I hardly noticed it since I would've expected that to cause more problems. I suppose this is because both of KF2's maps use the same textures and it's not often a lot of textures pile up and they're pretty small, but I would've expected this to be more disruptive. A big theme in my work lately is lots of bugs on top of bugs making it hard to tell what's what.

Finally, I've mostly been working on a memory solution. The first thing I did is see what happened when switching SOM's code over to using the SomEx.dll heap, which I guess may benefit from tech advancements on Microsoft's side, unless their advancements applied back in time to Windows 95 code. This is worth mentioning because it did speed up loading maps by about a factor of 2.7 times. That's with no disk IO happening.

In my project that got it down to 50 milliseconds. That seemed high to me at the time, but it's possible it would've been passable if I'd known I'd botched that background processing job. Still 50ms isn't good at all, and would only possibly go unnoticed because the frames are being triple of quadruple buffered separate from the real refresh rate. This is just the only way I've been able to hit a smooth frame rate with Windows because it's so shaky.

I wrote some code to calculate how much memory is needed for model instances and have it ready in advance so that it's just parceled out to the map transfer subroutine. In hindsight I feel like I should've just allocated all of the model instance data itself since I'm sure that's going to be desirable at some point, but it's a lot more complicated and the code around all of this is already a mess. It kind of has to be because you can't just reorganize SOM without more or less replacing it in full, and I prefer to work with it as much as possible. I'm also not sure how it manages instances is best, but it's certainly understandable to how it does it. It just allocates memory (pretty wastefully TBH--edited: i.e. unnecessary stuff) for every single spawn point, plus lots extra for items. Really in theory it only needs instance data for how many things are on the screen. But in practice it's not clear you can predict that and I've learned from this work how much a CPU can do in the span of a few milliseconds, and it's really not a lot at all... even by today's standards I'm surprised how puny a CPU is. In SOM's day it probably wasn't even an order of magnitude difference (material sciences have not come as far as people tend to think in this area when it comes to raw numbers... games are still pretty much smoke and mirrors) but what makes things tough today is hitting 60fps, which SOM wasn't designed for. It targeted 30 though. That's a bit more cushy. The PlayStation games were designed around 15fps, meaning streaming was probably less intense. Really I'm not sure loading entire maps at a time is the best design, but they're not that big, so they're still kind of chunks I guess.

I imagined just copying some megabytes across buffers would be pretty zippy but it's not really, the milliseconds accumulate. Anyway, with this pre-allocation strategy the speedup wasn't as much as I hoped. It now takes between 18 and 20 milliseconds to switch maps in KF2. Because the frames aren't locked into refresh rate you can't see this, but that's actually longer than 1 frame at 60 fps, so it's not ideal. In theory that's a gap frame. But I think in practice Windows is pretty good at picking up the slack, and part of that's because Windows is so unstable it kind of has to be. Larger hiccups happen from time to time just because of Window's irregularity, with background apps and so on. So I'm thinking this will be good enough for the demo I'm preparing. And I will work on doing even better after the code is better organized. I actually think only trying to have instances for things on screen is not a bad strategy, and that could be combined with the fade in effect to make it workable. I don't mind seeing some fade in, I kind of like it TBH. That could allow for not even loading the models themselves until required too.

simple machines forum

News:

Multi-Lingual Boards|

Public Discussion|

Land ahoy! Seamless transition in maps?

Author Topic: Land ahoy! Seamless transition in maps? (Read 8755 times)

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

Holy Diver

Holy Diver says,

simple machines forum

News:

Multi-Lingual Boards| Public Discussion| Land ahoy! Seamless transition in maps?

Author Topic: Land ahoy! Seamless transition in maps? (Read 8755 times)

Multi-Lingual Boards|

Public Discussion|

Land ahoy! Seamless transition in maps?