Architecture of the Windows Service LMDBService (updated)

Here is the new diagram:

Azure Container Instance

To run my container on ACI, I need a specific image :

FROM microsoft/iis:windowsservercore-ltsc2016

If not, it does not work. The error will be the OS version is not supported.

Envoyé de mon téléphone Windows 10

Windows API vs ISO C++

It’s been a while I am writing softwares for Windows using Win32 API and C runtime. But since some few years, I use ISO C++ and STL features more and more.

My best friends are :

• std::wstring for string management
• std::vector for container
• std::shared_ptr for memory management
• std::wostringstream for stream

For sure, there is a little overhead compared to C runtime routines but life is easy.

Envoyé de mon téléphone Windows 10

Making a POST call with JSON data using CPPREST C++ SDK

The source code here is the same as the C# version in the previous post. The client makes a simple call like that: 

	std::string value2 = "azertyuiopqsdfghjklmwxcvbn";
	std::string buffer = Base64Helper::base64_encode((const unsigned char*)value2.c_str(), value2.length());
	std::wstring value3(buffer.begin(), buffer.end());
	SetData(key, value3, value3.length(), dbname);

 

Here is the wrapper code for SetData. It’s included in HttpLMDB dll: 

bool HTTPLMDB_API SetData(std::wstring key, std::wstring valueb64, DWORD dwLen, std::wstring name)
{
	std::wstring port = Constants::MasterNodePort;
	std::wstring ip = ServerHelper::GetIP();
	std::wstring url = ServerHelper::BuildURL(ip, port);

	std::wstring contentType = _T("Content-Type");
	std::wstring contentTypeV = _T("application/json");
	std::wstring keepAlive = _T("Keep-Alive");
	std::wstring keepAliveV = _T("false");
	std::wstring contentLength = _T("Content-Length");
	
	std::wostringstream bufLen;
	bufLen << contentType.length() + contentTypeV.length() + keepAlive.length() + keepAliveV.length() + contentLength.length() + 4;
	std::wstring len = bufLen.str().c_str();

	std::wostringstream buf;
	buf << url << '/' << Constants::Request << Constants::VerbSetDataB64
		<< _T("&name=") << name;
	url = buf.str().c_str();	
	//{"key":"key_toto0","value":"value_toto0"}
	std::wostringstream bufjson;
	bufjson << "{" << '"' << "key" << '"' << ":" << '"' << key << '"' << ","
		<< '"' << "value" << '"' << ":" << '"' << valueb64 << '"' << '}';
	std::wstring jsonv = bufjson.str().c_str();
	//wcout << _T("jsonv : ") << jsonv << endl;

	http_client client_lmdb(url);
	http_request request(methods::POST);
	request.headers().add(contentType, contentTypeV);
	request.headers().add(keepAlive, keepAliveV);
	request.headers().add(contentLength, len);
	request.set_body(jsonv);

	http_response response;
	response = client_lmdb.request(request).get();

	wcout << response.to_string() << endl;

	return true;
}

 

Here is the code. It’s not very difficult.

Handling a long url using POST verb

To be able to store data in my LMDB Service, I need to store data as base64 items. To do that, I need to transmit json data from the client to the server. It can’t be passed on the url. So I use post handling. Here is the C++ handler:

void TheServer::handle_post(http_request message)

{
       try
       {
             g_Logger.WriteLog(_T("handle_post"));
 
             PrintRequest(message);

             std::wstring request = ServerHelper::FindParameter(message, _T("request"));
            
              if (request == Constants::VerbSetDataB64)
             {
                 RequestVerbSetData64(message);
                    return;
             }
             else if (request == Constants::VerbGetDataB64)
             {
                    // Does not work yet
                    RequestVerbGetData64(message);
                    return;
             }
       }
       catch (...)
       {
             // an internal problem occured
             g_Logger.WriteLog(_T("handle_post exception..."));
       }
 
       message.reply(status_codes::OK);
};

 

The C++ routine here to analyze is RequestVerbSetData64. Here is the code:

void TheServer::RequestVerbSetData64(http_request message)
{
       USES_CONVERSION;
       CLMDBWrapper lmdb;
       g_Logger.WriteLog(Constants::VerbSetDataB64.c_str());
 
       std::wstring dbNameW = ServerHelper::FindParameter(message, _T("name"));

       std::string dbName(dbNameW.begin(), dbNameW.end());
       std::wstring json;
       web::json::value jsonV = message.extract_json().get();

       Data data = Data::FromJSON(jsonV.as_object());
       TCHAR sz[255];
       _stprintf_s(sz, _T("Data key:%s value:..."), data.key.c_str());
       g_Logger.WriteLog(sz);

       if (lmdb.Init((LPSTR)dbName.c_str()) == false)
       {
             g_Logger.WriteLog(_T("LMDB Init not done !"));

            message.reply(status_codes::OK);
             return;
       }

       LPSTR lpszKey = W2A(data.key.c_str());
       LPSTR lpszValue = W2A(data.value.c_str());
       DWORD dwLen = strlen(lpszValue);

       lmdb.SetData(lpszKey, lpszValue, dwLen);

       message.reply(status_codes::OK);

       lmdb.Uninit((LPSTR)dbName.c_str());
}

 

The source code is simple to write, simple to read. Because it is native code, it is fast and we just need to distribute the dll we use. here, it’s just the C runtime, the C++ runtime and the CPPREST dll. This is the advantage of the native stuff, you don’t need to distribute any framework that size is around 350 MB… It’s lightweight, it’s fast, it’s built on the metal.

Running LMDBService WS NoSQL in Azure Container Instance

Now that LMDBService runs in a docker image, it is possible to run it into Azure Container Instance.

Here are the steps to follow:

• Build the docker image locally with its dockerfile
  • docker image build –tag mydocker/myserver d:\dev\docker
• Create a Repository in Azure Repository
• Connect in docker shell to the repo
  • docker login lmdbwsprod.azurecr.io -u lmdbwsprod -p xxxxxxxxxxxxxxxxxxxxxxxxxx
• Tag the prod repository
  • docker tag mydocker/myserver lmdbwsprod.azurecr.io/prod
• Push the docker image to the Azure repository
• Go to Azure portal and choose Azure Repository
  • Choose prod repository
  • Check tags and choose latest
  • Click Run Instance in […] button
  • Open port 7001
  • Click OK
• Go to Azure Container Instance
  • choose the container and click Overview
  • Retrieve the IP address

Test the URL : http://40.114.209.198:7001/MyServer/LMDB/?request=set-data&key=Key_v99&value=Value_v99&name=cache_NET

 

Architecture of the Windows Service LMDBService

Here is it:

Running a Windows Service with LMDB as a REST Web API Web Server in a Docker container

To make the test, I use the Microsoft/IIS image available at : https://hub.docker.com/r/microsoft/iis/

This image is managed by Microsoft and regulary updated. Size is around 1.8 GB. To get the image in Docker, run:

• docker pull microsoft/iis

Then, we to create a Docker file to instruct Docker how to setup the environment :

FROM microsoft/iis
COPY *.* c:/
RUN sc create LMDBService start=auto binpath=”C:\LMDBService.exe”
#RUN net start LMDBService
EXPOSE 80
EXPOSE 7001
RUN md c:\temp

Then, we need to build the image :

• docker image build –tag mydocker/myserver d:\dev\docker

And now, we can run the container as a deamon:

• docker run -d -p 7001:7001 -it mydocker/myserver

In order to test the Web Server in a browser, we need to know the IP address of the web server… We need to know the IP of the container… We begin by listing the containers :

• docker ps –a
• we take the first item of the list

And we run the following command using the id of the container:

• docker inspect -f “{{ .NetworkSettings.Networks.nat.IPAddress }}” 543e57f54047
• response is displayed => 172.26.255.203

Now we can test the web server in the container.

The container runs the windows service who host the web server… Let’s use this url :

• http://172.26.255.203:7001/MyServer/LMDB/?request=set-data&key=key_toto0&value=value_toto11110&name=cacheA2

It works !

 

LMDB – I love that stuff !

Since few months, I work on LMDB for Azure stuff. First, I have:

• migrated the library from Linux to Windows x64
• adapted the .NET Layer Interop
• made custom clients code

Now, I have setup the Azure Web App Svc for hosting a simple file uploader where files will be stored in LMDB NoSQL database. The way of using it is so simple :

        private static void StoreFile(string path)
        {
            string key = path;
            string value = String.Empty;
            byte[] buffer = File.ReadAllBytes(path);

            LMDBEnvironment _env;
            string dir = "c:\\temp\\cache_net10B";
            _env = new LMDBEnvironment(dir);
            _env.MaxDatabases = 2;
            _env.MapSize = 10485760 * 100;
            _env.Open();

            DateTime dtStart = DateTime.Now;
            var tx = _env.BeginTransaction();
            var db = tx.OpenDatabase("DB", new DatabaseConfiguration { Flags = DatabaseOpenFlags.Create });
            var enc = System.Text.Encoding.UTF8;
            tx.Put(db, enc.GetBytes(key), buffer);
            tx.Commit();
            db.Dispose();

            DateTime dtStop = DateTime.Now;
            TimeSpan ts = dtStop - dtStart;
            string str = String.Format("Time elapsed for set:{0} ms", ts.TotalMilliseconds);
            Logger.LogInfo(str);

            _env.Dispose();
        }

 

If you want to test LMDB in your environement, download that stuff:

Download the LMDB Windows DLL and console_app.Exe
Download the LMDB Windows DLL, the .NET WRapper DLL and console_app.Exe

Microsoft REST SDK Architecture

The architecture of the Microsoft CPPREST SDK is presented here for an article I write for french magazine Programmez.

It explains why the code is multi-platform. It’s based on Boost libraries. Boost is known to be a very good library, state of the art C++.