Eclipse Cloud

[Godlord]

@Minikloon:

> I renamed the thing to Eclipse Cloud, more representative of the plugin features with the current emergence of distributed virtual hosts and such.
> I also went with a more colorful design.

That sounds _way_ better.

@Minikloon:

> I'll be getting a new version out of Cloud Core (v0.3) with the content and configurations features. I'll also look into Stephen's comments in regard of the maximum number of connections Windows can handle using TCP and selector using an old plugin I wrote to simulate the Micro Engine (Added to the thread for download).

The maximum amount of connections has nothing to do with TCP or UDP specifically. The selector makes use of the select() call which uses FD_SET. FD_SET has different definitions for different platforms. On Microsoft Windows it is a linear array that has a predefined amount of entries (always 64 if nobody resets it). Even worse, if you actually do reset it, then you are just wasting time: its insertion and deletion time is O(n), and the select itself might be O(n). With any other platform select() is somewhat better, supporting 1024 sockets with better access times (I forgot them though, but you can probably find them on the web), but still, select() is something you should **not** use.

If you are going ahead with TCP through low-level sockets, then the amount of libraries to aid you with this task is zero. The only thing that is useful are BSD sockets and/or Winsock (for the low-level socket interfacing) and IOCP (Windows, Solaris and AIX) through WinAPI, epoll (Linux) and/or kqueue (BSD, FreeBSD, OpenBSD, NetBSD, Mac OS X). The last three libraries allow scalability, which is important when writing TCP servers. To give you an idea of how powerful scalability is: epoll can handle about 100K of connections with a system load of below 1%. And this model works best as an asynchronous TCP networking model that is supported by a thread pool.

The problem is that it isn't the best solution. UDP is generally faster to implement properly and works better when done right, and for UDP you actually have libraries that do all the work for you (e.g. enet and RakNet). With UDP you'll have to implement a protocol on top that deals with connections, RTT and that allows you to send unreliable and reliable messages, eventually sequenced or ordered (luckily RakNet and enet should both do this for you to some extent). Then that protocol can be used effectively because of its flexibility compared to TCP (which only supports reliable & ordered) and because of the performance gain you'll get.

There is a lot to consider when dealing with networking programming though: multi-threading is important, but scalability is usually way more important. It's also good to know your models, since some of them are extremely old and unusable due the fact that are way better and less limiting things out there nowadays. Not to mention that when you for instance deal with port forwarding, a new door has to be opened, since most networking libraries are not capable of dealing with UPnP, NAT-PMP or NAT punchthrough/introduction.

Then you also have the issue of writing servers for normal machines and writing servers for servers. Servers are aimed at running for a long time, therefore memory shouldn't be fragmented and should be used as less as possible. This is where your plug-in system just won't work, because it is too expensive to be run on such servers, and therefore it doesn't meet those requirements.

Regards,
  Stephan.

[Robin]

@Admirаl:

> TCP or UDP select() FD_SET. FD_SET O(n), O(n). elect() 1024 sockets BSD sockets Winsock IOCP (Windows, Solaris and AIX  WinAPI, epoll and/or kqueue (BSD, FreeBSD, OpenBSD, NetBSD, Mac OS X) epoll asynchronous TCP networking UDP (e.g. enet and RakNet) RTT  RakNet and enet TCP multi-threading scalability UPnP, NAT-PMP or NAT punchthrough/introduction.


[Minikloon]

Well Stephen,

I've tested reaching more than 64 connections on my Windows x64 Laptop and it worked well. I'll try with my Ubuntu 11 x86 later. If it gets stuck over time I'll look into working with enet.

Here's a screen at 63 connections :
http://imgur.com/y7cKM
And here's another at 67 connections :
http://imgur.com/shB02

Maybe my test isn't good in some ways. If it's the case feel free to tell me why and I'll try to find a solution. I've also created a topic on SFML's forum to demystify the issue (http://www.sfml-dev.org/forum/viewtopic.php?p=32791#32791).

I'm aware of the scalability problem the system might show once you get to hundred of players. But tell me, have you ever seen any Eclipse game with more than 100 players online?

Also, it would be possible to create a plugin that would route connections to other servers when connecting to a world, if anybody ever needs it.

[Godlord]

@Minikloon:

> Well Stephen,
>
> I've tested reaching more than 64 connections on my Windows x64 Laptop and it worked well. I'll try with my Ubuntu 11 x86 later. If it gets stuck over time I'll look into working with enet.

That is what it makes you think. Allow me to clarify what happened exactly.

@Minikloon:

> Here's a screen at 63 connections :
> http://imgur.com/y7cKM
> And here's another at 67 connections :
> http://imgur.com/shB02

Your server simply accepted all the 63, and eventually all the 67 connections. What you however did ignore is the fact accept() works on the server socket, and the server socket is likely to be the first in your FD_SET. When you actually send data with your 64th, 65th, 66th or 67th client, then you mightn't be notified, which is exactly the problem I am talking about. [Here](http://msdn.microsoft.com/en-us/library/ms740141%28v=vs.85%29.aspx) is the MSDN article about the select() function, which is internally used, which says that FD_SETSIZE is set to 64 (cfr. remarks). And [here](http://tangentsoft.net/wskfaq/advanced.html#64sockets) is yet another link to confirm that issue. And [here](http://www.sfml-dev.org/forum/viewtopic.php?p=32791#32791)'s the final one.

@Minikloon:

> I'm aware of the scalability problem the system might show once you get to hundred of players. But tell me, have you ever seen any Eclipse game with more than 100 players online?

No, I haven't, but I've neither seen a server that is scalable, except for the old EE server that used the IOCP DLL. And here's my problem with Eclipse: those who are creative and skilled enough, are limited by what Eclipse can. And if you are going the same way, then that be it, but it's definitely not the right way to think, in my humble opinion.

@Minikloon:

> Also, it would be possible to create a plugin that would route connections to other servers when connecting to a world, if anybody ever needs it.

I really do wonder if you have ever heard of the phenomena named 'DLL hell' or 'dependency hell', it describes exactly why plug-in systems require much more management. Creating a plug-in for _everything_ seems expensive to me, but if you think you can prove me wrong, then go ahead.

Regards,
  Stephan.

[Admiral Refuge]

@Admirаl:

> And [here](http://www.sfml-dev.org/forum/viewtopic.php?p=32791#32791)'s the final one.

LMAO, that just made my day!!

Also, Stephan, your name confuses the world ;D

[Robin]

@Admirаl:

> And [here](http://www.sfml-dev.org/forum/viewtopic.php?p=32791#32791)'s the final one.

I love you.

[Minikloon]

I've never heard of DLL Hell, probably because I've never experienced any such thing. Even after overloading Firefox and Paint.net with hundred of plugins.

> No, I haven't, but I've neither seen a server that is scalable, except for the old EE server that used the IOCP DLL. And here's my problem with Eclipse: those who are creative and skilled enough, are limited by what Eclipse can. And if you are going the same way, then that be it, but it's definitely not the right way to think, in my humble opinion.

I'll look into enet, its event-driven polling system looks attracting and simple.

Anyway thanks for your help, I'll release a quick version of the Core tonight with code cleaning and enet.

@Robin:

> I love you.

-image removed: bandwidth stealing-

[Helladen]

Good to see you guys changed the name, best of luck with the project.

[Devogen]

Apple corp. is going to love that new name! bravo.

Let's have a minutes silence for your imagination.

[mrmiguu]

@Devo:

> Apple corp. is going to love that new name! bravo.
>
> Let's have a minutes silence for your imagination.

Wait-
Did the name 'Cloud' originate from Apple or Microsoft? I first heard of it as 'The Cloud'.

[Minikloon]

I first heard about the name 'Cloud' from GOS, a now-dead Linux flavor that wanted to create an entirely virtual operating system. Then I heard about 'Cloud' from EyeOS v1, offering a similar service without the OS coding part. Then I heard about 'Cloud' from VPS.NET, offering Cloud VPS services. Then I heard 'Cloud' from Microsoft and finally, now, Apple.

Also, I'm the last one who cares about Apple Inc. iBuzzwords.

[SawQuart]

Keep the name. A lot of companies use "Cloud" now, ignore them.

[Devogen]

I was being a bit sarcastic, Apple like to claim everything…

As for the name, well it's better than Funeral thats for sure. Only downside is just sounds like jumping onto a popular word bandwagon and riding it.

[Minikloon]

Quick update.

I've finished the code cleaning and started a configuration handling embryo. I've ported 90% of my SFML Network Layer to RakNet. After working with RakNet for the first time, I see how higher-level it is while staying efficient.

I've updated the two download links. The release version has been name-updated, while the development version offers the latest modifications (not compilable right now, mind you).

There are still several other ideas to study about the Core.
The first one I had is concerning one of the issues I had with unmanaged messages. RakNet offers to create enums to describe our custom user types. I think it would be wise to create a message class and message handler for Cloud Core so plugins can create their own messages. All of that would be for the inner working of Cloud make the plugin development easier.

RakNet packets aren't done the right way for now because I didn't modify the plugins yet. For now they're all in the same struct type. I want to serialize my message class into bitstreams and profit of RakNet's fixed packets structure to use the less bandwidth possible server-side, isn't it a MMORPG engine anyway?  :embarrassed:

–

I've also thought about the client side. The client is another project, even if it follows the same plugin structure, it will give less liberty to the plugins but more power, I guess.

First off a Renderer. The client engine's renderer will be a pure virtual class so you can modify the rendering/graphic engine you wish without modifying the whole sources. The base class will offer windowing and drawing using general Sprite classes. This might be non-efficient because the renderer will need to create it's one kind of sprite from the Cloud's sprite class on each frame. I'll look into that issue.

Then there's a resources manager, quite simple meant to not load the same resource twice. I'll probably use a similar system as I used in the Micro Engine, with images IDs.

Configuration Handler, see Cloud's one.

Also there should be a GUI system of some sort. Maybe using the configuration handler to simply position stuff on screen, or any preferred idea.

--
_Meanwhile, in Canada…_

Btw, I'm not looking for any particular help. This is a development thread, so I'm posting my thoughts and development logs publicly so anyone can eventually participate by giving opinion without necessarily looking in the sources. Some might think I'm questioning the community about all of the software design, be sure that's not the case, I'm thinking out loud that's it.

[Prince]

I can't download , it says file does not exists or something.

Could you upload to Mediafire or Megaupload ?

[Minikloon]

I've updated the main thread with new Mediafire links, even if after testing the links again, they are working for me.

I'll find a reliable host.

[kibbelz]

when i extract the files it says they are currupt.
*edit*
It works now never mind :/

[dpinoy13]

:/
any screens?

[Minikloon]

Nao,

I've completed my port of my NetworkLayer to RakNet. I still need to test it out with a RakNet client. I didn't complete the configuration system yet, it's still in building, because I quickly switched from TinyXML++ to RapidXML, the latter offering parsing nearly as fast as simply loading the file. There's still testing to be done an iteration dev. No matter completion, that created a new dilemma about configuration data and how to use them.

I really want to keep configuration handling by the core, because otherwise every plugins would use their own way of handling configs and users would eventually be confused about where each configs are.

XML parsing is (relatively, ~) slow compared to direct text configs. Editing XML files might be easier for the users, but it's for sure way more enjoyable programming-side. What I think would be the best solution is that all core configurations would be done within a core.xml file. The Core would basically parse all .xml files, organization dictates me to use one file by plugin, but it would really not matter. But in the case the user wants another type of config handling, such as .ini like Eclipse Origins or even a DB like MySQL, he should edit the core.xml file to indicate, in some way, the configuration type infos. The solution is quite obscure for now.

I might also put the configurations on a higher level by using Boost.PropertyTree, still in study, depends of the final design decision.

There would also be the solution the put the configuration handling in a single plugin, but that would give more overhead to loading plugins, unless I implement some way of having managed messages between the plugins and the Core.

I'll need to test some stuff and possibilities out. Maybe increasing the plugin SDK size with some files isn't that bad…

Anyway I updated the Release and Dev versions with the latest versions.

* * *

Work on the Core will be enough for now, I really need to spend time on my client, setting up the engine features I want before starting to finally code appropriate plugins.

–
_Meanwhile, in Canada…_

Woot! I reduced the development zip size from 98 mo to a mere 1 mo by deleting useless VS2010 shit no-one cares about!  :embarrassed:

Also, I'll post tons of screenies when I'll have something exciting to show for the end-user.

[Minikloon]

Hai, really quick notifications about the client.

I've been working on the client today and I'm expecting a first release sometime before the end of the month. I'm trying to get basic features working and be scalable. For example, the input manager and the graphics engine can be easily changed, that means you can change my implementation of SFML graphics on the engine to your implementation of OpenGL, for example.

There will be a lot of testing to be done regarding of the plugin API, which by now isn't even attracting for me.

–
_Meanwhile, in Canada…_

http://www.ludumdare.com/  :cheesy:

[Minikloon]

I've putted up a download link for the current version of the client (See main post). It's a simple RakNet client built onto Eclipse Cloud's code. I'll need to pack in basic engine features such as an input handler, renderer, GUI system of some sort and more in future versions, based on needs and most logical requirements. I'll probably use SFML again for the work. For once I'll ship the code first before blabbing about the current state, so stay tuned with the client advancement.

–
_Meanwhile, in Canada…_
Advancing bits by bits I'd say, I'm trying not to stress myself too much with my open coding projects.

[Bloo]

Could you post the features of this engine?

[Minikloon]

> Quick features :
> Quote
>
>     - Easy to use plugin system.
>     - C++ means more efficient than VB6.
>     - Total control over your creation without the need of editing the sources.
>     - Interchangeable client.
>     - Open Source like any Eclipse (And free, as in free beer!)
>     - Multi-platform, runs on Linux without trouble.

First post.  :P

Besides,

I've uploaded the latest version of the software. The sources have been revisited to include meaningful comments. The client, badly enough, has been cutted from its engine features in work because it didn't compile.

[krankzinnig]

If you are still looking for a good way to do human readable config files I would suggest checking out config++. Its a really sophisticated yet tiny way to make great script-able config files with an almost C like syntax.

[Minikloon]

Hai.

I've updated the download link with the new download page (And the latest versions). I'm also working on a small updater to run on post-compile so you guys always have the latest development version. I'll look into updating the second post sometime.

I've mostly worked on the client, 'bout time isn't it? Most of the features are half-implemented badly enough…  :embarrassed: Well, I've completed the Resource Manager, inspired from some other sources. Basically, using templates and inheritance, you can create any resources you wish to fit with your graphics library, sound engine...

Resources are used by your implementation of graphics class. The Pic class (For Picture) has been created as of now and I'm trying to implement an SFMLPic. Works well, for now...

All of this is because of all the blabbing around DD8 implementation in Origins. Now it will take an (~)hour to convert Zap to OpenGL, SDL, DirectDraw X, whatever! In fact, I'll probably use the same resource model for the sound system.

I'm still thinking about a good Input Manager. I've read stuff and I'm not satisfied with it.

There's still a lot to be done relating to the plugin API, both server-side and client-side. And there's the whole client to be done. I guess I'll hit 0.3 when those two things will be done and **only then will I start doing plugins.**

Now, this is only a development release. Some may argue I'm not developing fast enough, but really, I'm doing as fast as I can. I've got things to learn, fun to enjoy, addicting forums to visit, porn to watch and even more stuff.

–
_Meanwhile, in Canada…_

I've bought the book Clean Code. A marvelous software engineering book that I'm madly devouring.

**Oh and btw, personal life is calling, I'll be out for a month without Internet connection nor computer. Original post modified on purpose. See ya back on 2011/08/13.**