SharpLife - Dot Net Core based modding p Created 6 years ago2018-05-06 19:15:02 UTC by Solokiller

I've been doing some research on porting Half-Life to C# using the Dot Net Core framework. I've got a working prototype that has C# code that can interface with the engine, though it's still far from complete.

I'm calling this SharpLife, since it's implemented using C#. It isn't limited to that language since it uses Dot Net but that is what i'm working with.

The goal is to port the SDK to C# and then upgrade it with Half-Life Enhanced's bug fixes and additions. Once that's done i can complete the planned changes for that.

Here's a diagram showing the overall design of the framework itself:Here are some examples of how it's easier to work with C# than C++:

[quote]
//Save restoring a field in a class in the SDK:
CBaseEntity* m_pGoalEnt;

static TYPEDESCRIPTION m_SaveData[];

TYPEDESCRIPTION CBaseEntity::m_SaveData[] =
{

DEFINE_FIELD( CBaseEntity, m_pGoalEnt, FIELD_CLASSPTR )

};

//Save restoring a field in a class in C#:
[Persist]
public CBaseEntity m_pGoalEnt;
[/quote]

//Getting the name of a class in the SDK:
const char* pszClassName = STRING( pEntity->pev->classname );

//Getting the name of a class in C#:
var className = pEntity.ClassName;

[quote]
//Loading an XML file in HLEnhanced:
https://github.com/SamVanheer/HLEnhanced/blob/346a9889f7da589f72cc66a71ee1202fc434714a/game/shared/CWeaponInfoCache.cpp#L154

//Loading an XML file in C#:
try
{

using (var stream = new FileStream("SharpLife/cfg/SharpLife-Wrapper-Managed.xml", FileMode.Open))
{
    var serializer = new XmlSerializer(typeof(ServerWrapperConfiguration));

var config = (ServerWrapperConfiguration)serializer.Deserialize(stream);

LoadMod(config.ModInfo);

return true;

}

}
catch(Exception e)
{

Log.Message($"{e.GetType().Name} - {e.Message}");
Log.Message(e.StackTrace);
return false;

}
[/quote]

It's much easier to manage dependencies than in C++ as well. To use XercesC in HLE you need to download, extract, compile and install it before referencing it in your CMake configuration.

In C# you only need to reference the NuGet package System.Xml.XmlSerializer. Once that's done everybody who checks out the codebase will be able to get the packages by restoring all NuGet packages for the solution.

For less experienced programmers (most modders) the language is easier to use as well. One mistake i've seen made a lot is string comparisons:

const char* somestring = ...;

if("ON" == somestring )
{
//Do something if ON
}

This compares memory addresses, which will always be false.

In C# it works like it does in most modern languages:

string somestring = ...;

if( "ON" == somestring )
{
//Do something if ON
}

And it does what you'd expect.

For scripting there's no need for Angelscript because you can just load assemblies that can directly access mod code. A simple plugin system would be a list of Dot Net assemblies that are loaded on mod startup that then behave as though they were part of the mod itself.

The language is very similar to Angelscript which is intentional, since Angelscript is based off of both C# and C++.

In addition, there's a scripting language called CSharpScript that is essentially C# code compiled at runtime, much like Angelscript: https://github.com/dotnet/roslyn/wiki/Scripting-API-Samples

By moving to this it'll be easier and faster to continue development, modders can focus on implementing game logic instead of working around language limitations and quirks.

It will take some time to implement, but right now the biggest issue is getting the native<->managed code interop working. I have the necessarily functionality done for that, so it's just getting the interfaces for the server done, then re-implementing everything piece by piece.

The priority is making sure the game can load and run with a minimal codebase, so most entities won't work yet for some time.

I'm hoping once that's done i can implement the physics code re-implementation, which will allow for parenting, and then if i have time and motivation implementing a new networking system to transfer data to clients. The client itself will also need to be converted, but it's a bit smaller than the server so that should be simpler.

Best case scenario i can skip the engine entirely and just do everything in C#, which would really make things easy but that's a lot of work and i don't know if i'll still have time and motivation for that.

Wow. Hope this works out epically.

Nice - this is something that I've wondered about for a while.
I guess it's possible but apparently not without its challenges.

Yeah, very cool!

Putting anything to do with Half life into a C# environment gets a big ol' thumbs up from me. Very happy to see C# get popular, Facepunch are doing a "gmod 2" by putting a layer of C# over UE4 for ease of use purposes. Would be great to see!

It looks like it's just the client and server code being ported - not the engine, so I imagine it's just plain old C# and the .Net standard library mostly.

I'm only using Dot Net Core with compatible libraries, and native interop to run under the engine. I do want to eventually implement engine code in managed code to skip the engine entirely but that's not going to happen any time soon.

I'm surprised C++ is so verbose and isn't equipped with a dependency manager.

I understand it's less verbose, easier to manage dependencies, and run on all platforms without recompiling. I hope it's not too much off topic, but why C# specifically? For instance, why not Java? (I'm just curious :p)

I can just write unsafe code to access model data, worst case scenario i can write my own loader and do what i've done in C++ before. It's not that hard.

Here's what i did to access edict_t: https://pastebin.com/DQ1qeXus

I haven't been able to test it yet, but it should work. This lets me access the native data while still using managed code in the game itself.

C++ actually does have a dependency manager, but as with most things C++ it's new and not terribly widely used yet: https://blogs.msdn.microsoft.com/vcblog/2018/04/24/announcing-a-single-c-library-manager-for-linux-macos-and-windows-vcpkg/

Even so it's still not easy to add dependencies, you still need CMake and old projects like GoldSource aren't built for it so you need ugly hacks to make it work.

I chose C# because it's the best tool for the job. The language itself is very easy to use and lets you design simple interfaces, like you can make a property that returns a string that internally could do a lot of work without exposing that detail. In Java you'd need a method for that.

You could do this:

//In BaseEntity
[Persist]
public string ClassName => pev.ClassName;

//In entvars_t
//Data is a pointer to the native entvars_t instance
//ToString accesses the string pool containing native string data
public string ClassName => Data->classname.ToString();

If i can ever get the engine out of the equation that can be simplified to:

[Persist]
public string ClassName { get; private set; }

In Java you can't do that. They've been playing catch-up to C# for a few years now, but C# is definitely the better language and the tooling support (Visual Studio) is much better than Eclipse/IntelliJ.

Native interop in Java isn't nearly as powerful as it is in C#. I read that they're planning to implement C++/CLI like support in Dot Net Core sometime, so perhaps next year i can simplify the interface even more by writing the native access in C++ instead. That would make things even easier, though by that time i may already have made the native parts obsolete.

That part's a bit tough, since there's a lot of engine code that accesses entvars_t and edict_t. There's 57 known accesses to entvars_t members that represent strings (string_t), the majority of which are classname checks.

Those can be avoided by handling entity creation entirely in managed code, since these calls are part of map spawning, entity creation, etc. If you override specific engine functions using the Detour framework then you can handle this directly in managed code.

If i can do that, and unregister certain console commands the engine won't touch it anymore. Eventually the idea is to not let the engine do anything, at which point the native interface would go away and the managed version would have its own implementation, or would be removed (edict_t and entvars_t really have to go).

I'm designing the game interface to make sense and be sane so most code wouldn't need updating. Direct access to edict_t and entvars_t would be handled in properties in common entity base classes (see example above), making the switchover easy and painless.

edict_t would probably be re-purposed to do what it does in Source, which is to store networking data and provide a way to represent players before their class instance is created (which i can fix if i have control over those particular engine functions).

If i can get this all done i can implement that custom networking i've talked about, which would let me implement uniform entity networking. It would look much like the persistence example:

[Networked]
public float Health { get; set; }

Should this not work and a notify-on-change type be needed, it wouldn't change the public interface:
[quote]
[Networked]
private Networked<float> NetHealth { get; } = new Networked<float>(0);

public float Health
{

get => NetHealth.Value;
set => NetHealth.Value = value;

}
[/quote]

All Networked instances in an entity would be hooked up to notify their owning entity so it can track the changes and network them at the next opportunity.

That would mean networked entities would exist on both the client and server side, could network any amount and type of variables and probably not cause reliable channel overflows anymore. cl_entity_t wouldn't exist anymore, the hacky weapon prediction system would be integrated into the rest of the system and we could actually build some neat things without worrying about engine limits.

Unfortunately this is all a lot of work and i have to finish up the native<->managed interface first, then port the entire server and then make the client version of all this. I can probably speed the client conversion up by writing a code generator for the interfaces, since it's largely the same thing over and over.

I'd still have to write a new GUI library either way since interfacing with VGUI(2) is not an option here. Any OpenGL based library should do here, as long as i can take the result and put it where VGUI goes it'll work, but there aren't many for Dot Net Core right now. I should be able to use the VGUI2 interfaces to render directly to the engine here though.

As far as content development goes, things should be a lot easier for you. I can implement a plugin system that loads assemblies that reference the mod, letting you access the game's code directly so you could add new weapons that way.
That is probably one of the biggest problems with Sven Co-op's scripting system, and one people have asked me about a lot, so i'm happy that this problem is now solved.

If i can implement the server<->client bridge i could add a debug UI that can directly access the server to take care of things that would otherwise be done through restrictive console commands. More expansive AI and scripting debugging would certainly help.

It's a bit too early to go into details about that, but i'm hopeful that this will let me finish the planned features for HLE and beyond.

If you're having issues loading data or rendering it then it's not caused by the language, that's the library you're using or the code you've written. I've had issues with that in C++ as well, it doesn't make C++ crap.

Java's way worse than C#. I've worked with it for years and it is missing a lot that C# has, and i wouldn't use it for something like this. C# has much better native code support than Java which by itself is already a reason to use it here because i need to do a ton of marshalling and native code access.

Package management is pretty much automatic in C#, whereas Java doesn't have that. I wouldn't want people to juggle dependencies all the time, which is a problem that HLE is suffering from due to the CMake based approach requiring a lot of manual building of dependencies.

My college is even switching their courses from Java to C# because it's much better and easier to teach and use.

Like... I'm struggling to learn and and apply a programming language (currently mainly using Java) to make a living, and many times I feel like an impostor.

But after seeing this...*goes out for a walk Escobar style

Java and C# are MOSTLY similar when it comes to the syntax, the main differences are the keywords ("final" -> "const", "package" -> "namespace"...) and other minor things. The transition from Java to C# should be fairly easy.

Okay your decision with C# I hope you have to prepare OpenGL-Wrapper like OpenTK ,OpenGL.net etc...

Good luck! Headcrab Basher -

I won't have to touch OpenGL for some time, there seem to be half a dozen libraries available so i'll take a look when it's time to work on it.

Like... I'm struggling to learn and and apply a programming language (currently mainly using Java) to make a living, and many times I feel like an impostor.

But after seeing this...

This is really advanced stuff that most programmers will never have to do, so don't be discouraged by it.

Java and C# are MOSTLY similar when it comes to the syntax, the main differences are the keywords ("final" -> "const", "package" -> "namespace"...) and other minor things. The transition from Java to C# should be fairly easy.

C# can do anything that Java can do, but Java can't do everything that C# can do.

Events, properties, directly calling delegates from native code, Java either can't do it or is much worse at it. I'd much rather be using C# than Java if i had a choice.

I've implemented the Detour framework so i can improve performance in some areas. Entity creation was really slow so i hooked it and re-implemented it in managed code. Mods can now control the entire process, it allows the creation of templates that can be spawned with a point_template like entity.

This means there is no maximum length limit on keys and values, no limit to the number of keyvalues that an entity can have and no limit to how many entities are defined in a map, although the engine's limit still applies during map spawn time.

I've also implemented support for linking classes to their map name, handling multiple names (some weapons need this), and partial support for keyvalues declared as properties, fields or methods.

I'll be implementing engine functions so i can implement the game interface, which will allow networking and such to work. Then i can implement some entities so maps work and i can see what else i need to implement before it's functional.

I'll probably need to detour more engine functions for performance reasons, which means physics will be re-implemented in managed code as well eventually. That also means parenting support.

Saved games will also need to be detoured to avoid problems with saved games being too big since the maximum size is hard-coded, which will allow the game to control all of the data being saved.

I'll get all of this stuff on Github soon so i can show the code. It's not ready for use yet and the framework seems to be triggering Windows Defender in a way that causes it to use up to 6 GB of memory at times. I don't know why this happens but it may be because it detects a custom CLR host.

It also seems to be causing crashes on startup (before my code is even loaded) so i'll have to figure out how to prevent this before it can be released.

Once upon a midnight dreary there was 8-bit. It was invented of the technological limitations of its day. As technology grew more powerful and advanced, so did everything around it. Graphics changed and polycounts increased 20% by the year. Yet the diehards stay behind and give their favorite titles the love they deserve before they get buried in the sands of progress.

Along comes the indie developer. Mr. Indie doesn’t have the tools, talent, processing power, or even the man power to compete with AAA title quality. He lives by the only rule he knows; do what you can, with what you have, right here, right now. And thus he studies his 8-bit ancestry. There is a fundamental difference between then and now as he creates his game. He has better tools and far more computing power than his ancestors. With this, he creates with less limits, so his game naturally has less technical limits. He doesn’t have to consider palettes, or color count, z-fighting, or fitting his game into a 2mb cartridge. Out of a lack of necessity, a new style had been created, this is high-bit.

It is time we see what we can do with goldsource with better tools, better computers, and without limits. It’s come a long way so far, and it still has a long way to go. Let’s high-bit this bitch.

I've put the source code for the wrapper and the game on GitHub:
https://github.com/SamVanheer/SharpLife-Wrapper
https://github.com/SamVanheer/SharpLife-Game

There is no documentation on how to use it yet (you'll need it), and it's still in development so it's subject to change.

Here's the list of changes since my last post:
-No more crashes on startup, unknown what caused it but i haven't experienced it anymore. Defender still uses a ton of memory though

-Implemented a bunch of entities, the training map can be loaded and it has triggers, doors, ladders, letting you move to the end of the map. No changelevel yet, that's partially implemented but it's missing the code to transition entities, which is part of the persistence system

-Player physics is implemented and seems to be mostly working, with the exception that movement on ladders seems to have some issues. There might be missing code in some other area, i'll look into that

-Most of the base classes have been implemented, aside from logic that handles bullets in BaseEntity everything up to BaseToggle is finished

-multi_manager and multisource have been implemented and support an unlimited number of targets now

-You can register cvars that are integrated into the engine, it's nothing fancy but it lets you work with it

-Entity data loading works properly and can handle most inputs quite well

-With the exception of networking code nothing accesses EntVars directly, it's all done through properties which will make switching over much easier

-Edict is avoided wherever possible, most references are in code that accesses the engine and some are wrapped to reduce the number of direct uses

-The node graph and func_train use EHandle to reference entities, which makes the code more reliable since it can't refer to other entities by accident

-The materials file is loaded, all supported materials are defined and contain all relevant data defined in one place: https://github.com/SamVanheer/SharpLife-Game/blob/3af4cda9fc5ec3ed74fd0df38268a41cf6883ab8/src/Server/Game/Materials/MaterialsSystem.cs#L29

Currently hard-coded, but this can be updated to load materials data from config files. It eliminates most of the redundant code and simplifies the logic when it's used

-The sentences file is loaded, its behavior is slightly different but should still be compatible with the engine's version. Once the client is implemented i can handle it there and remove some of the backwards compatibility stuff

-Unused code has not been converted, there are unused functions, class members, as well as logic meant to protect against engine behavior that has since changed. This simplifies things a bit

There are random crashes in the engine that happen from time to time, i'll have to debug that and see what exactly is happening.

I've looked into existing persistence support in the Dot Net framework, it seems that the best solution in this case is to use XML serialization. This will be mostly automated and should work like the original, but without the complicated binary logic.

Here's an example of how an entity is defined in the SDK and the C# version:
https://github.com/ValveSoftware/halflife/blob/5d761709a31ce1e71488f2668321de05f791b405/dlls/plats.cpp#L628
https://github.com/SamVanheer/SharpLife-Game/blob/3af4cda9fc5ec3ed74fd0df38268a41cf6883ab8/src/Server/Game/Entities/Plats/FuncTrain.cs#L22

If you look through the code you'll see some patterns. Spawnflags are defined as a static class named SF that contains constants, that makes it easy to find and reference them.

I've looked into the engine's save game code, it looks like i should be able to override it by handling a few key functions, but i'll need to grab an engine data structure to initialize it properly when saved games are loaded.

I've also looked into the physics code, most of it is handled by one function: https://github.com/id-Software/Quake/blob/bf4ac424ce754894ac8f1dae6a3981954bc9852d/WinQuake/sv_phys.c#L1507

I can take the code from Quake and modify it to match what the engine does, then i can start moving properties from EntVars to BaseEntity. That'll improve performance and reduce memory usage a bit.

Key physics code is only referenced outside of that function hierarchy and engine functions in the player physics code, which i can also override.
That code is what converts edicts to physents, so i might be able to convert all of that to BaseEntity as well at some point.

I won't be able to do that until i've got control over the client as well, since it uses the same code.

Once i do have the client done i can make a networking system to override the engine's version. Then i can start overriding the big stuff like precaches and entities. That's probably going to take some time though.

The big question is whether i should focus on framework stuff like that, or implement all of the game's code first. The former lets me get the entire framework up and running, but leaves it unusable as a modding platform until it's complete, the latter gets it operational but limits functionality to what the SDK can do, with a few additions.

I think the former is the best solution, since it lets me get the client done sooner as well, but if anybody has an opinion, i'm all ears.

Former definitely seems like the best option, if you rush to support operationality people will observe it as sort of "Spirit of HL 2" rather than fully realise the potential of what's at hand. Feel like people need stuff to work and play with from the get-go rather than trickle down over time.

Having an initial framework also allows the opportunity for there to be decent documentation from the get-go, something which UE4/UDK seriously suffered without in its infancy.

I've been working to get the client operational, i've got everything up until the managed wrapper done, now it's time to get the actual game code ported over.

Unfortunately i've hit a problem: SDL2 names its threads in a way that causes the game to shut down if it's launched with a managed debugger attached.
This behavior can be disabled, but not in the SDL version used by Half-Life. I've contacted Valve and asked them to update the SDL version, but i doubt they'll do anything about this.

This could also explain the seemingly random crashes i've experienced, if a thread is started and named it would cause the game to exit immediately if ran with a managed debugger so there's that. I'll keep an eye on the exit code to verify this.

Fortunately i can avoid the issue on startup by launching without a debugger and then attaching it, but it is a pain in the ass. Eventually i can ignore the engine and then this problem will go away, but for now it's an issue.

When i ran it without the debugger it seemed that it works without crashing, the player's view is stuck at world origin and nothing is rendered because visibility testing and physics isn't present yet, but that should be relatively easy to implement.

Once i've got that done, i can start working to add a networking layer to build on top of. That'll be the basis of the engine replacements, starting with sending large amounts of data, such as weapon and ammo types, sending a copy of the precache lists, etc. I've got some ideas about optimizing networking that could help but that remains to be seen.

I'll also be looking at available Dot Net Core GUI libraries, there aren't many from what i've heard so i may have to build one. It only has to be able to do what VGUI(2) can do, so a custom one wouldn't be that large. As long as it can render to an OpenGL texture/buffer it should be good.

I've implemented enough to load a client game project now, it's using the same player physics code as the server but nothing's rendering now as before.

I'm considering merging the server and client libraries since they'll be sharing a lot of code, including entities. It makes sense to use the same code with server and client specific providers for things like networking to make things easier, it enforces good design and separation.

I know Minecraft does this, mods are both client and server capable but you can tell it whether to enable the interaction on the client and server sides.

If i were to do this i could probably optimize singleplayer servers by not networking anything and using the same entities for both client and server. This would reduce memory usage and time spent "networking" (sending over loopback) things but could cause changes in behavior when data that is not sent at 100% accuracy is accessed on the client.

I'll have to think it over some more, i could just as easily use a library that shares entity code but one library means it's all in one place. That could be very useful for things like tools which could load the library and query for any data that they need, for instance it should be possible to generate FGD files with the metadata present in the server library.

Right now i'm going to add the code needed to render the game world correctly (telling the engine that entities are visible so it renders them), and then making sure input is handled. This code will need to be updated if i move away from the engine's code but having a working implementation should make it easier to convert.
That way any bugs added during conversion can be fixed without needing to look through my own version of the engine's code.

If all goes well i should have a functioning - if bare bones - C# based modding platform by the end of the week.

I think that this is an awesome project.

Always happy to see somebody else working on a project idea that's so ambitious it seems impossible, but slowly coming up with results.

I experimented with Protobuf a bit to see how easily i could make a messaging system:

//Tell the client to change its name to Solokiller
var message = new StuffText
{
    Text = "name \"Solokiller\"\n"
};

//Represents the server's outgoing message buffer for the given client
var outboundPacketStream = new MemoryStream();

//Queues this message for sending to the client
//First we add a list of messages so the receiving end knows what's coming, and how to parse the messages
var messages = new MessageList
{
    Messages =
    {
        new Message
        {
            //The Id is not defined as an enum in a .proto file because we have both server-to-client and client-to-server messages
            //Using the same enum is a waste of bandwidth (variable length encoding would be larger for no reason) and duplicating the message class is wasteful
            //This cast is needed in only 2 or 4 places (sending messages, receiving them) depending on whether we'll use a generic handler type or one that's hardcoded to the Id type
            Id = (int)ClientCommandId.StuffText,
        }
    }
};

//Write the list, then each message
messages.WriteDelimitedTo(outboundPacketStream);
message.WriteDelimitedTo(outboundPacketStream);

//The client's buffer for processing
var inboundPacketStream = new MemoryStream();

//Would be a net packet send to the client, read by the client into a buffer for processing
outboundPacketStream.WriteTo(inboundPacketStream);

//WriteTo sets the position to the end, so this is needed to read from the start
inboundPacketStream.Position = 0;

//Read the message
var messagesResult = MessageList.Parser.ParseDelimitedFrom(inboundPacketStream);

//Process each message
foreach (var messageResult in messagesResult.Messages)
{
    //Could make a dictionary mapping id to a type containing the parser and a handler method to process the incoming message
    switch ((ClientCommandId)messageResult.Id)
    {
        case ClientCommandId.StuffText:
            var result = StuffText.Parser.ParseDelimitedFrom(inboundPacketStream);

            Console.WriteLine($"Command is: {result.Text}");
            break;
    }
}

So this code makes a StuffText message, sends it, parses it on the "client" and gets the text.

The Quake equivalent looks like this:
https://github.com/id-Software/Quake/blob/master/WinQuake/host.c#L322
https://github.com/id-Software/Quake/blob/master/WinQuake/cl_parse.c#L800

This approach is safer and easier to use. Messages are sent with their size so malformed messages will never cause problems (svc_bad is an indicator of this sort of problem), you can also do validation by marking fields in the .proto file.

Protobuf has no overhead, all it sends is the message data. I'm using the delimited write/parse methods here so a variable length integer is prepended to each message, but that's necessary because the parser wouldn't know where messages end otherwise. If you were to convert all network messages to use this it would be more efficient because Protobuf can encode data more efficiently, for instance you can send entity indices with fewer bits if it's a low number since it can use variable length encoding.

You can read more about encoding here: https://developers.google.com/protocol-buffers/docs/encoding

Eventually the idea is to let you send messages to clients by creating these messages and passing them to the engine's networking system, somewhat like this:
Networking.SendMessage(client, myMessage);

Where Networking is an instance of an object that can send messages, SendMessage is a method that can send to a specific client, client is the target client (index or client instance, probably the former) and the message itself.

Adding new messages is easy, the engine's current limitations won't be a problem here (192 byte maximum for user messages).
It would probably look like this:
Networking.AddMessageType(typeof(MyMessage), MyMessageHandler);

Where MyMessage is the generated class that it can get the parser and metadata from, MyNetworkingClass is the class where you handle your incoming messages and MyMessageHandler is a method with this signature:
void MyMessageHandler(MessageData<MyMessage> message)

Where the MessageData type contains the message itself as a member Message, along with anything else that's relevant (probably nothing, but it allows for changes).

The ability to remove and re-add handlers should also be possible to allow for changing the handlers for cases such as multiple HUDs, where some handlers will depend on the current HUD instance to process it.

So the idea behind this is to make networking a lot easier. Variable length encoding and packed fields will reduce message size, optional fields makes it easy to handle messages with multiple options (e.g. SVC_TEMPENTITY has over 100 possible options, some of which have optional members), and because it's based around code generation it's all strongly typed. No more screwing up a temp entity message because you sent a byte as a short or anything like that, it'll tell you if you made a mistake like normal programming.

It's also possible to send messages to programs that use a different language, so this could be used to send messages between C++ and C# if the need arises. I doubt that will actually be needed, but this could be used to send messages between the game and any tools, like Hammer. That's necessary if the tool is running as its own process, in which case inter-process communication is needed, which makes sending data a lot harder since you can't just pass data around like you normally would.
You could for instance serialize objects to protocol buffers to pass them around, though for that JSON might be a better solution.

It's also easy to log entire messages for debugging since the message size is known, as is the list of all messages in a packet. You could use this information to dump an entire packet for analysis if the game detects something is wrong (e.g. svc_bad errors, assuming those can happen with this design).

The next step will be to look into a good networking system to use, then build something that combines these two things.

I've looked at a few networking libraries and it looks like Lidgren is the one to go for: https://github.com/lidgren/lidgren-network-gen3
It's recommended by most and provides everything that's needed to replicate the original system.

It also supports DotNet Core, albeit requiring a manual build. It should be possible to distribute the NuGet package itself and load it up locally, as i've done. That's relatively simple to do compared to the C++ dependency management approach.

Unlike GoldSource Lidgren doesn't impose any packet size limitations, but hardware and software limitations in the network infrastructure still apply here. Once a packet grows beyond a certain size it will be fragmented and reassembled on the other side, which means high amounts of packet loss can still cause problems. I may have to look into creating something that can send data in discrete packets to limit the effects of packet loss, but that will require further investigation to see if it's a real problem or not. I'm not too familiar with that stuff so it may not be a problem.

In any case it means the "reliable channel overflow" errors should be a non-issue. Timeouts, high ping and excessive packet loss should be the only reason to disconnect clients.

As far as GUI libraries go i've found that Imgui has a .NET Core wrapper: https://github.com/mellinoe/ImGui.NET
This should make implementing debug overlays simple, i know suXin from Facepunch has done this before so it should be simple to implement.

There's also a library for graphics programming: https://github.com/mellinoe/veldrid

It's low-level, cross platform and supports multiple backends, so i think this is a good way to support Vulkan and D3D.
It uses SDL2 for window management, and the library names are the same as Half-Life's so there is a need to modify the source. This is because Half-Life uses an older SDL2 library that may be missing functions. I may need to make a custom build for this to work out. I see that Veldrid uses a library to load native libraries, i may be able to provide a custom SDL2 path to load it.

For keyboard and mouse input i can also use SDL2, but i'll need to use a .NET wrapper here that also works together with Veldrid.
The wrapper i found uses DllImport which may not cooperate with manual library loading: https://github.com/flibitijibibo/SDL2-CS

Normally any given library is loaded once, so the goal will be to make sure the right SDL2 library gets loaded. If i can control where it looks for native libraries i can use a custom one and bypass the entire issue, otherwise i'll need to modify the source code of all involved libraries.

One option is to use this: https://msdn.microsoft.com/en-us/library/system.runtime.interopservices.defaultdllimportsearchpathsattribute(v=vs.110).aspx

But that only affects the assemblies that it's applied to. What i'm thinking is i can do this: free the original SDL2 library, load the correct one in the engine assembly, marked with that attribute, and then rely on the other load calls to use that library. This depends on the SDL2 library used by the original engine being freed, which may not be possible since it's a load time linked library. I'll have to verify this, otherwise using renamed libraries is the way to go.

That should cover networking, (debug) graphics and input. Sound can be handled through FMOD, that has a C# wrapper. The engine doesn't use it so there's no problem there.

I already have the filesystem code written up, though some changes are needed here and there. I'll be ditching the registry part of its initialization since that forces it to have Windows only dependencies, it'll be stored in a general purpose engine config file, along with things like the default game directory ("valve").

I've also built a command system, i've taken into account the problems that Valve had with engine upgrades to the classes. The problem there was that the class layout couldn't be altered without breaking mods, i've solved this by not making modders create instances of the classes. Instead, you provide info objects that describe commands, which are then used to construct the classes. Combined with C#'s delegates allowing class member functions and the compatibility between libraries, it should solve all of the problems that the command system has had since Quake.

As an example, here's the alias command:

RegisterConCommand(new ConCommandInfo("alias", arguments =>
{
    if (arguments.Count == 0)
    {
        Console.WriteLine("Current alias commands:");
        foreach (var entry in _aliases)
        {
            Console.WriteLine($"{entry.Key}: {entry.Value}\n");
        }
        return;
    }

    // if the alias already exists, reuse it
    _aliases[arguments[0]] = arguments.ArgumentsAsString(1);
})
.WithHelpInfo("Aliases a command to a name"));

Compare with the original: https://github.com/id-Software/Quake/blob/master/WinQuake/cmd.c#L341

The registration methods return interfaces to the commands to allow you to reference them, add more handlers, directly query variables, etc.
Commands and variables both allow you to add multiple handlers, so for instance if you have your cheat cvar "sv_cheats" defined somewhere, you can do this anywhere:

var svCheats = commandSystem.FindCommand<ConVar>("sv_cheats");

svCheats.OnChange += (ref ConVarChangeEvent conVarChangeEvent) =>
{
    if (!conVarChangeEvent.Boolean)
    {
        players.ForEach(p => p.DisableCheats());
    }
};

This adds a handler that checks if cheats have been disabled, and if so disables cheat properties on all players.

Variables also have filters that are checked before their values are changed. It's a pretty simple thing, if the filter returns false the change is not allowed. Filters can also modify inputs to for instance clamp values to a range, similar to Source's cvars. I've added filters that can do a few basic things, like testing if inputs are in a list, allowing only numbers, inverting filters, etc. For example, sv_cheats can have a boolean filter applied that restricts all inputs to 0 or 1.

Filters can be added and removed as well. Since both filters and change handlers are events, removing handlers you don't have a reference to is impossible, at least unless you use reflection to get the underlying event handler. It is thus difficult to accidentally break existing handlers.

Once all of this stuff is set up, the essentials should all be there. When i can draw stuff onscreen debugging will be easier, and i can start writing libraries that can load the various file formats used by the engine (bsp, mdl, spr). These will be standalone to allow for re-usability, hopefully useful to make new tools with.

If i can find the time i should be able to set this stuff up soonish, but i can't promise anything. I'm hoping there won't be any more unexpected problems, but you never know with these sorts of workarounds.

The new SharpLife engine is running under Half-Life as a mod now, having taken over execution. Now all further development should be done in C# only.

The engine codebase is now on Github: https://github.com/SamVanheer/SharpLife-Engine

I've worked around the SDL2 issues by renaming the library used by the managed libraries. This means there are 2 SDL libraries being used.
It doesn't appear to be causing problems so far though.

EDIT: spoke too soon, using multiple libraries causes problems because the engine creates a window as well, the library attaches data to the window which both libraries recognize so it ends up calling into the wrong one. I had to change the source code so it uses a different tag so it doesn't conflict.

There appears to be a bug with SDL2 on Windows 10, i emailed Valve about it: https://github.com/ValveSoftware/halflife/issues/1887

This should also let me use the same library, thus solving the problem entirely for this project.

This seems like a pretty interesting port, it would be cool to see where it goes in the future. Also, is this a port of Xash or regular GoldSource?

Technically it's neither. Xash is illegal and porting GoldSource would also be illegal. It's a mod that replicates the engine's behavior without using original GoldSource engine code.

If it doesn't requires the Steam client, it's illegal.

If it doesn't requires a Steam account with ownership of Half-Life 1, it's illegal.

If it doesn't requires Half-Life 1 Steam version to be running, it's illegal.

SharpLife doesn't violate any of the above legal restrictions so it's fine.

Got the SDL2 window up and running. Had to patch the SDL2 library again because more strings were conflicting with the original. I hope Valve updates the library soon.

So for those of us who aren’t as clued in to the programming side of things (so, any of it), in broad terms, what’s the state of this project so far, and what’s to come?

It's still pretty early on in development, it won't be anywhere near ready for months.

It will have an OpenGL 3 based backend, which means better graphics performance and better effects if you want it, increased/eliminated engine limits, better networking, better debug overlays, better entity management for programmers. Limits that depend on fixed size buffers are now unlimited as long as there's no need to network the data.

There's a lot that i can't predict can be done until we get there but on the whole it's better than the original.

Solokiller, you never cease to blow me away. Good luck man haha

I've been working on getting Veldrid integrated, unfortunately i've encountered a showstopper bug: https://github.com/mellinoe/veldrid/issues/100

Basically Veldrid doesn't work in 32 bit applications (and it should), so i can't get this to work until that's fixed. I hope the problem can be found soon, but there's no telling where the problem is.

Beyond that i can do some work writing libraries to load BSP, MDL, SPR, etc file types to make things easier down the road.

The problem has been solved, i can now continue implementing the renderer.

Veldrid's working, along with ImGui on top. I'll need to simplify the code since the sample code is way too much for what's needed right now, but it's a good start.

Now i can start adding ImGui based menus and debug output to make things easier, and load 2D and 3D data to render it. That represents a sizeable chunk of what's needed in the engine.

Sure let me just write an entire GUI library before i can start drawing stuff on-screen.

ImGui is meant for debugging, not actual UI development. It's easy to use, fast and there's already a library for it.

If you need to show blocks of code use pre tags: https://twhl.info/wiki/page/TWHL:_WikiCode_Syntax#wiki-heading-16

Finished the essentials of the renderer, added in skybox rendering.

Framerate's pretty good, should probably clamp it though.

What are the advantages or additional capabilities that expand upon what we already have as users?

Nothing yet, it's mostly backend stuff. I'm about to start implementing texture management, that's where things will change a bit. The original engine has a limit on the number of textures, how large they can be, etc. There's no reason why there would be any limitations here.

Veldrid uses ImageSharp for texture loading, no other libraries are needed.

Well i like it.

I set my projects to copy local assemblies so all of the required files are in the assemblies directory. It's a simple solution.

C# is managed code, so there is no need to make a native executable, not that it would matter in this case anyway since the executable is the original Half-Life executable which i can't replace. My code is all libraries, so combining them isn't an option either way.

The command you're using uses a Mono tool to consume a .NET Core executable, obviously that will never work.

@SourceSkyBoxer The only reasons .NET Framework is used nowadays:

• You need to make a Windows only desktop application.
• You need to make a Windows 10 UWP application (like the Windows Store).
• You need to use a NuGet package or library that doesn't exists yet in .NET Core.
• You are using technologies not yet supported by .NET Core (mostly Visual Basic, F# and some ASP .NET stuff).
• You are developing on a platform that doesn't support .NET Core.

And that's it, so why .NET Core is better and it's gonna be the new "C# programming meta":

• It's cross-platform: it works on Windows, Linux and Mac. Something that .NET Framework doesn't do.
• It's faster and optimised: if your application is strict on "performance budgets", this is way better than using .NET Framework.
• It has "microservices" in mind: enterprises will love this for sure when migrating big applications.
• It integrates nice with Docker containers: because a lot of people love Docker for building, testing and deployment and it's used by many continuous integration services.
• It's open-source: everyone can see the code and contribute to it unlike .NET Framework.

In a nutshell, Mono is dead, it only survives because of Unity/Xamarin, from what I saw, making a GUI application with Mono is a pain in the #!$. They had their chance to take over Microsoft, but now it's too late.

.NET Framework is gonna die soon as well, not as fast as Mono, but it will eventually when missing features from .NET Framework will be added back.

net standard 2.0 is shit because it can't bundle as single executable. So sadly who forgets requiered net assemblies than it "can't work".

Read this documentation from Microsoft, and especially read the "--self-contained" part, it basically does the same thing as your "mkbuilder" but for .NET Core applications:

https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-publish?tabs=netcore21

UPDATE: ninja'ed by Solokiller

NET Core runs on Linux and Mac natively, it doesn't run through WINE, it doesn't compile to native code like Mono does.

The --self-contained option packages the runtime along with the exe, but it doesn't embed assemblies into the exe. For reference, SharpLife references the default 32 bit installation directory at this time, though it may eventually change when i get to testing the Linux version.

Alright i think we've talked enough about Mono, this project uses NET Core so all that's irrelevant anyway.

Progress update:BSP & WAD loading is finished, currently rendering the entire map but the code can be adjusted to use VIS data pretty easily.

Now i'll have to make sure the camera angles are correct so movement is properly translated and everything, since at the moment the world is actually sideways.

I wrote the code to load both file types.

I've only completed these two things, i haven't even started on entities.

You have admirable patience to keep engaging him, Solo. Endlessly contrarian is the impression I've gotten.

I think i'll stick to porting Quake's code and updating it to work like GoldSource does it, no need to start breaking stuff by using physics code from some book.

Reminder that Half-Life SDK code is not compatible with the GPL2 (Quake 1/2 source releases) and that projects like Xash3D, that violate those two licenses by mixing them have had disapproval by Valve.

So if you can (if you don't use Valve code directly), change the license on your code that's on git repo to be licensed under something GPL compatible instead.

That is, if you want to use Quake code.

Valve doesn't approve of Xash because they used leaked code, not because of the license. People have used Quake code in tools and mods before with no problems from Valve. Even Sven Co-op has some code taken from Quake in it, and that was before they got the engine.

Even if they don't approve, if they don't send a cease and desist it doesn't matter anyway. Projects like ReHLDS have been going for years doing things far worse than mixing the two codebases with no legal issues.

Your lack of care and knowledge in regards to software licenses is worrying.
Valve may not care about these projects but the Free-Software-Foundation and Zemimax might.

Valve doesn't approve of Xash because they used leaked code, not because of the license.

Sure, Xash3D may use leaked code in addition to GPL and HL1 SDK code. That doesn't change the fact that the GPL prohibits merging code with a license like the one in the Half-Life 1 SDK. This is not just a concern for Valve, but also Zenimax and the FSF.

Even if they don't approve, if they don't send a cease and desist it doesn't matter anyway.

It may not matter to you, but it matters to anyone that is using your software (which is what this is all about, right?). Some may depend on it and get screwed later down the line because you don't actually own the rights to the code.

I know people who actually dealt with Zenimax in regards to the Quake engine license (This was for Xonotic's standalone release). They still license it (for about 10 grand) and they'll negotiate a license with you. However they'll gain ownership over any changes you make and you start off with the original Quake 1 sourcecode release (you can't expect to buy a license and use something like the Darkplaces engine, because of the GPL code).

They have dealt with C&D's over the past decade, in both the Quake and Doom community. Don't push your luck.

I'm not using Quake code directly, i'm referencing it to rebuild parts of GoldSource. I can't change the license because i'm using code from the SDK to rebuild parts.

You won't find actual Quake code in this because it's C code and pretty outdated. You wouldn't be able to to see any matching code as a result since it's still completely rewritten.

EDIT:

We've talked this over and we've concluded that no actual Quake code is in use at all, and the Quake license states this:

These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it.

SharpLife is based on GoldSource, and is largely designed from scratch, with reused code from the SDK.

From now on i won't reference Quake anymore when working on this project, so there will be no legal issues with that codebase.

I've taken this opportunity to make a list of systems that are needed to make a working GoldSource engine, and how Quake relates to it:

• Platform specific code used for window, input, graphics management: Provided by SDL2
• Platform specific code used for file input/output: provided by NET Core APIs
• Common code (tokenization, string utility functions): Tokenization code comes from multiplay_gamerules.cpp, string utility functions are provided by NET Core APIs
• Math functions: Provided by NET Core APIs
• Graphics: Provided by Veldrid, GoldSource specific code derived from public SDK code and completed by me
• Sound: Provided by FMOD SoundSystem library
• Memory management: Quake memory management is largely obsolete even in modern Quake derivatives, and C# doesn't allow that kind of control anyway
• File loading: Derived from SDK code
• Networking: I'll be using a networking system built using a NET Core library and ProtoBuf, completely different from Quake's QSocket and Quake 2's NetChan (used by GoldSource). GoldSource's networking code is pretty terrible so reusing any of it is a waste of time and effort anyway
• Various engine and game logic functions: derived from SDK code (parts of Quake engine code were moved into game code in Half-Life), missing parts can be re-implemented

Anything else is minor and can be reproduced without the use of Quake code.

Also of note: Quake is C with some assembly, which can't be used in C#. GoldSource has a C engine and largely C++ game logic. Quake code would have to be heavily modified to even be usable, to the point where it has to be redesigned from scratch to be useful. Since Quake and GoldSource rely on global variables in many areas, any properly designed C# equivalent would be drastically different.

Thus, no licensing issues.

Quake is C with some assembly, which can't be used in C#.

Code conversion from one language into another happens very often.
Besides, the ASM is only the sped-up 256 color software-renderering code so that's really irrelevant.

GoldSource has a C engine and largely C++ game logic. Quake code would have to be heavily modified to even be usable, to the point where it has to be redesigned from scratch to be useful.

I beg to differ. The game-logic from Deathmatch Classic, although it may be C++, is actually converted from the Quake 1.06 progs code.
A language that doesn't even have a concept of integers.

If Valve didn't own the licensed Quake code, that'd be, for example, a breach - if it was based off the GPL-released code and they didn't adjust their license properly to comply.

and the Quake license states this:

What you quoted is from the paragraph about modification of GPL2 code.
This does not allow you to take Quake code and put it into non-GPL works under any circumstances. It applies to your own modifications and when you can relicense them as something else. The license clearly states that once your code depends on working with GPL code, your code has to, I quote: "be on the terms of this License"

Thus, no licensing issues.

As long as you don't use any Quake code (converted, reformatted, copy&pasted etc.) you'll be fine.