nlib
nlib Main Page

This manual is a collection of libraries that support Windows, Linux, FreeBSD, OS X, Cygwin, and Nintendo's development environments.

About nlib

nlib is being developed to provide a group of user-space libraries that enable you to program for Windows, Linux, FreeBSD OS X, Cygwin, and Nintendo's development environments using the same interface. Its development follows a software-centric approach, adopts appropriate and pragmatic coding rules (nlib Coding Rules), and implements an interface that is independent from the individual development environments. This focus makes it possible to develop in parallel for multiple development environments while maintaining high quality, performance, and reliability.

Features Implemented by nlib

The main features implemented are listed below.
  • Reading and writing XML and JSON
  • Highly-efficient memory allocators
  • Test framework compatible with Google Test
  • Wrapper libraries that facilitate code development that supports both SSE4.1 and NEON
  • Succinct data structure libraries for features like keyword searching
  • UTF-8 reading, writing, normalization, and conversion from and to UTF-16 and UTF-32
  • Threads, mutexes, condition variables, TLS, barriers, read-write locks, and atomic variables
  • Getting and freeing system memory, getting the time, and reading and writing files
  • Functions and macro definitions for base C linkages used to provide multi-platform and multi-compiler support

nlib Operating Environment

Currently, compilation and operation is supported for the following environments.
  • For Windows:
    • Visual Studio 2008, 2010, 2012, and 2013 (32-bit static library builds, including Express Editions)
    • Visual Studio 2012 and 2013 (64-bit static library builds, including Express Editions)
    • Visual Studio 2012 and 2013 (64-bit DLL builds, including Express Editions)
    • For the newest 32-bit version of Cygwin: gcc 4.9.3
  • For Linux:
    • Ubuntu 15.04 (64bit): gcc 4.9.2 and clang (ver 3.6, libstdc++)
    • Ubuntu 14.04LTS (32bit/64bit): gcc 4.8.2 and clang (ver 3.4, libstdc++)
    • Ubuntu 12.04LTS (64bit): gcc 4.6.3 and clang (ver 3.4, libstdc++)
    • CentOS 7.1 (64bit): gcc 4.8.3 (libstdc++)
    • Fedora 22 (64bit): gcc 5.1.1 (libstdc++)
  • For FreeBSD 10.2 (64bit): clang 3.4 (libc++)
  • For OS X (10.10, Yosemite): Xcode (version 6.4)
  • For CTR: armcc compiler
  • For Cafe: ghs compiler
Depending on the compiler version and other factors, compilation uses C++11 features as appropriate (C++11 Features Used by nlib).

nlib Support Policy

Please contact the site where the package was released for instructions about how to get support. Also read the documentation, including the FAQ. This document and the code samples explain the current implementation. Future development plans and release dates will not be documented or made public. Support and bug fixes will be made in the latest version at that time. Special versions will not be provided for specific projects, and no special support will be provided. This rule is essential to keep the size of the development team in check.
Unannounced changes that break compatibility may occur if there is no way around it. Note this fact.
Before use, it is probably a good idea to perform some acceptance tests to determine performance criteria including whether the package works as expected, is sufficiently fast, and has memory consumption within the acceptable limits. Creating acceptance tests up front also helps to avoid trouble when upgrading nlib. It is a good idea to use the testing library to create these types of acceptance tests.
The bug may be quickly fixed if you describe the code reproducing the bug as a test code in the nn::nlib::testing format and include it in your bug report.

nlib Package Description

Note that the included libraries and supported platforms vary depending on the package.

Directory Structure

The package is structured as follows.
├── bin
│   └── cmake                 # Command-line tools
├── cmake                      # Common Cmake scripts
├── documents-en
│   └── API                   # Reference Manual
│       ├── html              # HTML version of the Reference Manual
│       ├── nlib.chm          # HTML Help version of the Reference Manual
│       └── searchdata.xml    # Search index
├── include                    # Include files (in `system/include` for Cafe)
│   ├── nn
│   │   └── nlib
│   │       ├── exi
│   │       ├── heap
│   │       ├── msgpack
│   │       │   └── jsonrpc
│   │       ├── oss
│   │       ├── simd
│   │       ├── succinct
│   │       ├── testing
│   │       ├── threading
│   │       └── unicode
│   ├── lz4.h
│   ├── lz4hc.h
│   ├── lz4frame.h
│   ├── sqlite3.h
│   └── sqlite3ext.h
├── lib                        # Platform-specific library files (`system/lib` for Cafe, `libraries` for CTR)
└── samples                    # Source code for samples
    ├── cmake
    └── sources
        ├── exi
        ├── heap
        ├── misc
        ├── msgpack
        ├── oss
        ├── succinct
        └── testing

Platform-Specific Descriptions

Descriptions of Each Library

misc Library (libnx_misc.a, libnx_misc.so, nx_misc-vc120amd64.lib, nx_misc-vc120dllamd64.dll, etc.)
This library comprises components that absorb differences among platforms, as well as collections of small-scale libraries. It implements many features that are believed to be commonly used in programs.
Specifically, it includes the following.
  • Functions that have C linkages for absorbing differences among the various environments (defined in nn/nlib/Platform.h)
    • Getting the date and time
    • Allocation and deallocation of (virtual) memory from the operating system
    • pthread-like threads
    • pthread-like TLS, mutexes, semaphores, condition variables, barriers, and read-write locks
    • One-time thread-safe execution (corresponding to pthread_once)
    • Atomic memory-replacement functions
    • Debugger access
    • Functions for file handles and file I/O
    • Search-related string functions optimized using SIMD instructions
    • String encoding conversion between UTF-8 and UTF-16 and between UTF-8 and UTF-32, in both directions
    • Implementation of printf-type functions that absorbs differences between environment-specific printf implementations
    • Basic lock-free data structure and the algorithm
    • Macros that absorb differences among the various attributes for each compiler
  • Base stream classes that serve as a foundation for InputStream and OutputStream
  • Classes that read and write UTF-8 strings from streams while validating (TextReader and TextWriter)
  • Classes that read and write binary data from streams (BinaryReader and BinaryWriter)
  • Classes for handling Unicode
  • Classes and functions for handling threads
  • Classes and functions for handling date and time information
    • Classes for adding and subtracting dates and times, in addition to parsing and creating RFC 2822 or W3C DTF date strings (DateTime and TimeSpan)
  • Classes and functions for handling URIs
    • URI parsers (Uri)
    • Classes for processing RFC 6570 URI templates (UriTemplate)
    • A class for standardizing file paths, which are often represented by different strings depending on the environment, into URI strings (NativePathMapper).
  • A class for handling asynchronous file access (threading::AsyncFileIo)
  • The different types of gadget classes
    • A substitute class for std::unique_ptr (UniquePtr). In C++11, this class is defined as std::unique_ptr using a typedef.
    • Singleton classes (Singleton and SimpleSingleton)
    • An std::vector-like class that does not require freeing memory and copying elements upon expansion (Nlist)
    • A class for parsing command-line strings (Nflags)
    • CRC (Crc32)
The namespaces include the global namespace, nn::nlib, nn::nlib::unicode, and nn::nlib::threading.
exi Library (libnx_exi.a, libnx_exi.so, nx_exi-vc120amd64.lib, nx_exi-vc120dllamd64.dll, etc.)
A library used to read and write binary XML. It can also read and write text-based XML. It supports XML namespaces and includes a simple, easily programmable pull parsing API.
The namespace is nn::nlib::exi. It depends on the misc library.
heap Library (libnx_heap.a, libnx_heap.so, nx_heap.lib, nx_heap-vc120dllamd64.dll, etc.)
This library is a heap library for which memory allocation and deallocation is optimized with multithreaded operations. The nmalloc and nfree functions (and the heap::CachedHeap and heap::CentralHeap classes used to implement them) keep performance degradation in check by controlling fragmentation through the use of multi-level free lists. By maintaining per-thread caches for small blocks of memory, memory can be allocated and freed without locking, which helps to limit the degradation of multithreaded performance. Pointers within the free lists are checked for both mangling and their allocation status. This implementation makes it more difficult to use them in an invalid way by overwriting memory externally.
The namespaces are nn::nlib::heap and the global namespace (C linkage functions). It depends on the misc library.
succinct Library (libnx_succinct.a, libnx_succinct.so, nx_succinct-vc120amd64.lib, nx_succinct-vc120dllamd64.dll, etc.)
A library used to handle succinct data structures. You can think of a succinct data structure as one whose content can be searched while it is still compressed.
The succinct library includes the following.
  • Implementation of succinct vector classes
  • A class that implements balanced parentheses
  • A class that implements LOUDS (level-order unary degree sequences)
  • Trie implementation
  • A class for creating high-compression indices and searching strings using the Aho–Corasick algorithm
The namespace is nn::nlib::succinct. It depends on the misc library.
msgpack Library (libnx_msgpack.a, libnx_msgpack.so, nx_msgpack-vc120amd64.lib, nx_msgpack-vc120dllamd64.dll, etc.)
A library for reading and writing MessagePack and JSON. It also includes a class for parsing CSV.
The msgpack library includes the following.
  • Classes for reading and writing JSON from streams
  • Classes for reading and writing MessagePack from streams
  • A class for high-speed in-memory MessagePack parsing
  • A class for reading CSV
  • Classes for storing JSON, MessagePack, and CSV data after it has been read
  • Implementation of JSON-RPC
The namespace is nn::nlib::msgpack. It depends on the misc library.
testing Library (libnx_testing.a, libnx_testing.so, nx_testing-vc120amd64.lib, nx_testing-vc120dllamd64.dll, etc.)
A library for writing highly portable test code. It is a testing framework that is somewhat source-code compatible with tests written for Google Test.
The namespace is nn::nlib::testing. (It is aliased to the testing namespace for the sake of Google Test compatibility.) It depends on the misc library.
oss Library (libnx_oss_bsdiff.a, libnx_oss_sqlite3.a, libnx_oss_lz4.a, etc)
Contains code for open-source software (OSS), code derived from OSS, and code for accessing OSS. To use it, an OSS license must be processed.
The following libraries are currently included. They are built as separate library files.
  • A library conversion of bsdiff that has been modified to use zlib instead of bzlib2, and the corresponding command-line programs
  • A version of SQLite3 that has been modified to run using the interfaces provided by nlib, and a corresponding command-line shell
  • LZ4 is approximately 20 times faster at data compression than zlib, and approximately five times faster at decompression.
The namespaces are nn::nlib::oss and the global namespace (C linkage functions). It depends on the heap library and the misc library.
simd Library (Mostly Inline Functions Included in the misc Library)
SIMD instructions can significantly increase the speed of programs. The simd library makes it possible to write code that supports both SSE4.1 and NEON.
It supports integer and single-precision floating-point number calculations that enable your code to compute at high speeds.
Single-precision floating-point number calculation can be used even in environments that do not support SSE or NEON.
On Cafe, this is optimized to use paired singles.
In other words, the same code can support the following environments.
  • On x86 processors, the code is compiled with speed boosts through the use of SSE 4.1.
  • On processors with NEON support, the code is compiled with speed boosts through the use of NEON.
  • On Cafe, the code is compiled with speed boosts through the use of paired singles. (Floating-point operations alone are quite a bit slower than the Cafe-SDK for operations like matrix calculations, because the interface returns values for data like four-dimensional vectors.)
  • On CTR, generic code is compiled (floating-point operations only).
The namespace is nn::nlib::simd.

C++11 Features Used by nlib

nlib is compiled using C++11 features for some combinations of compilers and versions. The table below summarizes the C++11 features, compatibility macros that can be used with nlib, and the compilers that compile using C++11.

Note that for clang, the determination is made based on __has_feature and __has_include macros, and not the version.

C++11 features are not used in any other environment.
C++11 Feature Identification Macro Macros and Typedefs for C++03 Compatibility Visual Studio gcc
Rvalue references NLIB_CXX11_RVALUE_REFERENCES None 2010 4.3
Variadic templates NLIB_CXX11_VARIADIC_TEMPLATES None 2013 4.7
Static assertions NLIB_CXX11_STATIC_ASSERTIONS NLIB_STATIC_ASSERT() 2010 4.3
Null pointer constant NLIB_CXX11_NULL_POINTER_CONSTANT nn::nlib::nullptr, nn::nlib::nullptr_t 2010 4.6
Deleted functions NLIB_CXX11_DEFAULTED_AND_DELETED_FUNCTIONS NLIB_DISALLOW_COPY_AND_ASSIGN() 2015 4.4
Explicit conversion operators NLIB_CXX11_EXPLICIT_CONVERSION_OPERATORS NLIB_SAFE_BOOL() 2013 4.6
Alias templates NLIB_CXX11_TEMPLATE_ALIAS None 2013 4.7
constexpr NLIB_CXX11_CONSTEXPR NLIB_CEXPR 2015 4.7
char16_t, char32_t NLIB_CXX11_NEW_CHARACTER_TYPES nlib_utf16_t, nlib_utf32_t 2015 4.4
alignas NLIB_CXX11_ALIGNMENT_SUPPORT NLIB_ALIGNAS() 2015 4.8
Explicit virtual overrides NLIB_CXX11_EXPLICIT_VIRTUAL_OVERRIDES NLIB_OVERRIDE 2012 4.7
final NLIB_CXX11_EXPLICIT_VIRTUAL_OVERRIDES NLIB_FINAL 2012 4.7
noexcept NLIB_CXX11_NOEXCEPT NLIB_NOEXCEPT 2015 4.6
Range-based for NLIB_CXX11_RANGE_BASED_FOR None 2012 4.6
lambda NLIB_CXX11_LAMBDAS None 2012 4.7
Delegating constructors NLIB_CXX11_DELEGATING_CONSTRUCTORS None 2013 4.7
std::unique_ptr NLIB_CXX11_UNIQUEPTR nn::nlib::UniquePtr 2013 4.7
#include <random> NLIB_CXX11_STDLIB_RANDOM None 2013 4.8
#include <chrono> NLIB_CXX11_STDLIB_CHRONO Conversion to and from nn::nlib::DateTime 2012 4.7
#include <atomic> NLIB_CXX11_STDLIB_ATOMIC Such as NLIB_MEMORY_ORDER_RELEASE 2012 4.8
#include <array> NLIB_CXX11_STDLIB_ARRAY None 2012 4.5
#include <unordered_map>
#include <unordered_set>
NLIB_CXX11_STDLIB_UNORDERED None 2012 4.5
#include <tuple> NLIB_CXX11_STDLIB_TUPLE None 2012 4.5

nlib Coding Rules

nlib Basic Coding Rules

The nlib coding rules are a slight modification and relaxation of the latest version of the Google C++ Style Guide (and a slightly outdated Japanese translation).
The reasons for using the Google C++ Style Guide are listed below.
  • It is easy to access documentation about the coding rules. Translations are available (albeit unofficial ones).
  • Similar to the general coding rules for the gaming industry, C++ exceptions and RTTI use are prohibited.
  • It has been updated continuously in the past.
  • Changes made to the coding rules with each update are reasonable, and there haven't been any sudden, drastic changes.
  • We can expect it to be maintained even five years from now.
The changes and relaxations adopted for nlib are shown below.

Checking and Correcting Coding Style

The coding style can be checked easily by using cpplint.
The warning criteria have been slightly relaxed, as follows.
  • Specify --linelength=100 to allow up to 100 characters per line.
  • Warnings due to cpplint's legal/copyright, build/include_alpha, and whitespace/newline categories have been suppressed.
  • Includes of the algorithm header are not required when using std::swap.
You can run checks easily by writing rules like the following in your makefile.
cpplint:
-./cpplint.py --linelength=100 \
$(shell find ./include/ -path "*.h") \
$(shell find ./sources/ -path "*.cpp" -or -path "*.h") 2>&1 | tee cpplint.log
In addition, nlib supports checking source code using clang-check. Generating a makefile with CMake creates a file called compile_commands.json, which clang-check uses to run.
For more information, see JSON Compilation Database Format Specification in the clang manual.

Automatic Formatting of Coding Style

You can also use clang-format to standardize the indentation and other stylistic aspects to suit your preferences. You can use the configuration used by nlib, which is stored in a .clang-format file (for clang 3.5) that is included in the nlib package. It is recommended that you first convert your source code en masse using clang-format, and then use cpplint.py to do common detailed checks.

Setting Warnings During Compilation

Code quality can be improved by testing with multiple compilers and runtime environments. Configure all compilers to relatively strict warning levels.
  • Ensure successful compilation with Visual Studio 2008 and all newer versions. In general, resolve all warnings shown when using the /W4 option.
    • Suppression of pragma warnings is permitted.
    • Make sure that the latest updates are always applied, and check _MSC_FULL_VER in your code.
    • Periodically reduce warnings using code analysis in Visual Studio.
  • Make sure that compilation succeeds using the latest versions of gcc and clang binary packages available for Cygwin (32-bit) and Ubuntu. In general, resolve all warnings shown when using -Wall -Wextra.
    • Always use the newest version of Cygwin. If a Cygwin update causes a build not to work as expected, revise the build scripts.
    • Suppression of pragma warnings is permitted.
    • Use -Wstack-usage=65536 to ensure that the stack is not overused.
    • Use -fstack-protector.
  • Make sure that compilation succeeds without warnings in the CAFE development environment.
  • Make sure that compilation succeeds in the CTR development environment.
  • It is preferable to have compilation succeed in all 64-bit environments.
  • Also resolve all Doxygen warnings.
The basic release conditions for nlib are ensuring that there are zero cpplint.py warnings and that compilation and tests succeed in all environments.

Additional Rules and Conventions

  • In general, code for functions that are not declared or implemented in functions defined in Platform.h must appear within code that can be compiled in all environments it could conceivably be used in. That said, inclusion of code that is optimized for specific hardware using SIMD instructions or other such techniques is permitted.
  • Use macros like NLIB_FINAL or NLIB_OVERRIDE and typedefs to safely incorporate C++11 features and continuously make it possible to port to C++11 or C++14.
  • Use POSIX.1 2008 error codes for error values whenever possible. Use the errno_t type, as recommended by https://www.securecoding.cert.org/confluence/x/QgAy.
    For example:
    typedef int errno_t;
    Using code like this makes it easier to understand that these are error values.
    • Values of POSIX.1 2008 error codes are sufficiently small integer values, so immediate values can be loaded efficiently on processors like ARM.
  • They must be implemented without using exceptions or RTTI. Make sure that your application code (that calls nlib) is written in a way that anticipates the use of these features, such as exception safety.
  • Instead of functions like malloc and free, use functions like nlib_malloc and nlib_free.
  • Use new (std::nothrow) for new.
  • The global new operator must not be overloaded within the library.
    • This overloading is not allowed because it affects the behavior of other libraries that you wouldn't think of as having dependencies.
  • Encode multi-byte strings in either ASCII or UTF-8.
    • The implementation must allow UTF-8, including in Windows. Path strings and strings passed to nlib_printf are passed to Win32 API after conversion to UTF-16 in nlib.
    • Command-line argument processing (Nflags) is an exception.
  • The char type must work correctly, whether signed or unsigned.
    • These can be handled explicitly as signed values by using signed char.
  • UTF-16 is supported if wchar_t is 16-bit; UTF-32 is supported if wchar_t is 32-bit.
    • NLIB_WCHAR_SIZE is defined as either 2 or 4, so you can use this to make the determination.
  • Use nlib_utf16_t as the type for UTF-16 characters, and use nlib_utf32_t as the type for UTF-32 characters.
    • In C++11, these types are defined with typedef as char16_t and char32_t, respectively.
  • In general, prioritize UTF-8 over UTF-16 and UTF-32.
    • UTF-8 consumes less memory, and is easy to support on multiple platforms.
    • The existence of surrogate pairs and combining characters make it impossible to always represent a single character with a fixed length, even if using UTF-16 or UTF-32.
    • Using nlib_strcplen_ex allows you to request the number of code points and byte length simultaneously while performing validation. If SIMD is supported, this operation can be sped up even more.
  • Move constructors and move assignment operators must be defined in code separated by NLIB_CXX11_RVALUE_REFERENCES, and you must also define member functions like the following so that the same functionality can be implemented in C++03.
    Type::Type(Type& rhs, ::nn::nlib::move_tag); // Behaves like the move constructor.
    Type& Type::assign(Type& rhs, ::nn::nlib::move_tag); // Behaves like the move assignment operator.
    Excellent alternative approaches like boost::move exist, but they cannot achieve full compatibility, so they are not used.
  • type_traits are used with care. This feature of C++11 is sometimes buggy or is not a complete implementation for some compilers or compiler versions. A tentative wrapper has been provided in nn/nlib/TypeTraits.h, but only use it if there is no other alternative.
  • Do not use non-copyable mixins.
    • Use NLIB_DISALLOW_COPY_AND_ASSIGN instead.
    • The increase in levels of inheritance can sometimes have a negative effect on source code analysis programs and documentation systems.
  • For the C standard library headers, include files in .h format. There are two reasons for this.
    • It handles situations when the C++ standard library headers are corrupted.
    • It minimizes code confusion caused by the inclusion of both <xxx.h> and <cxxx> formats within nlib. (The <xxx.h> format must be included because there is a requirement that nn/nlib/Platform.h can be compiled with a C compiler.)
  • Write tests in Google Test format, and make sure unit tests have been run on all platforms.
    • This makes it easier to find bugs that only occur in certain environments, in addition to platform-specific implementation bugs.
    • It is best if you separate the common tests to be run on any platform from the PC-specific tests.
      • This makes it easier to write stress tests and brute-force tests in environments with as much power as a PC.
      • On a PC environment, it is sometimes easy and convenient to use open-source software (OSS) in test programs.
      • In a Windows environment, DirectX and other such APIs can be used in test programs, which is sometimes convenient.
      • In environments where Google Test does not run, the testing library can be used.
      • When compiling using gcc, you can calculate test coverage automatically by using gcovr.
  • Write the Reference Manual in Doxygen format.
    • Doxygen is continuously being improved, and it is a documentation system that we can expect to exist and be improved even five years from now.
    • Doxygen comments are written within manual.h files located within each include directory. English versions of the manual are stored as manual_en.h. This approach allows multiple languages to be managed within the same repository.
  • If possible, enable building using CMake.
    • CMake is a representative selection for cases when you are coding in C/C++ that must run on both Windows and Linux.

Introduction to CMake

CMake can be used for building samples. CMake has been well-known for being adopted to build MySQL (MariaDB). Currently, it has established the position as the standard build system for various software supporting both Windows and Linux because it is written in C/C++. You may have noticed that many OSSs contain the file CMakeLists.txt. Using CMake allows you to generate makefiles for GNU make, Visual Studio and Xcode projects by writing this CMakefiles.txt script file.
The main advantages of using CMake are as follows.
  • It supports many systems and compilers.
    • It supports Windows, Linux, Mac OS X, FreeBSD, IRIX, AIX, Sun OS, HP-UX, and more.
    • It supports multiple IDEs, including all the Visual Studio versions, Eclipse CDT and Xcode.
    • You can also use Express Edition of Visual Studio and each of the latest versions of Visual Studio will be ready to be supported by the time when its RC version is released.
    • It can support cross-compilation with relatively simple settings.
  • Temporary files and other files created during builds don't clutter up the directories where the source files are located. This means Out-Of-Tree builds are possible. This function is always required when binary for multiple environments needs to be generated from one source tree.
  • Multiple independent projects can be generated from CMake and then used, which makes parallel builds easy.
  • It searches for the locations of required libraries and tools automatically.
  • The include path settings for libraries and links to libraries have been made cross-platform, so it is possible to write unified code even if the paths vary for each environment.
  • Installers can be created (CPack).
    • NSIS, deb, rpm, tgz, and so on.
  • Tests can be run and managed (CTest).
With nlib, using CMake allows you to generate solution files that work with all Visual Studio versions, makefiles for UNIX and Cygwin, and so on, and then build the program.
The ./cmake/toolchain-cafe.cmake and ./cmake/toolchain-ctr.cmake files within the package are configured for cross-compilation. This makes it possible to use CMake to generate makefiles even for CAFE/CTR sample programs, and then build the programs. The CAFE and CTR versions of nlib also have scripts for the standard make and omake build systems.

Cautions When Using CMake

  • A Windows version of CMake must be included to generate Visual Studio projects.
  • A Cygwin version of CMake must be installed to generate makefiles for Cygwin.
  • With CMake, Visual Studio and Xcode projects and makefiles must be output to separate directories for each Debug build and Release build.

Automating Setup of the Build Environment

To update software installation using the command line, you can use chocolatey in Windows or apt-cyg in Cygwin. Script-based automation is easy because it can be run on the command line. These tools are similar to apt-get in Linux. Providing scripts that use these tools makes it easy to set up the build environment.

Using chocolatey

The chocolatey tool can be installed by running the following command from a command prompt.
@powershell -NoProfile -ExecutionPolicy unrestricted -Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin
This makes it possible to download and install programs from the command line.
Examples are shown below.
cinst powershell4 # Install PowerShell 4.0.
cinst git.install # Install Git for Windows.
cinst cmake # Install CMake.
cinst doxygen.install # Install Doxygen.
cinst GraphViz # Install GraphViz.
cinst VS2013.4 # Install Visual Studio 2013 Update 4.
cinst VS2012.4 # Install Visual Studio 2012 Update 4.
cinst windows-8-1-sdk # Install Windows SDK 8.1.
cinst 7zip.commandline # Install the command-line version of 7-Zip.
The code below allows you to perform other operations on packages, such as updating and uninstallation.
cup all # Update all installed packages.
cuninst [packageName] # Uninstall a package.
cver all -localonly # Verify the installed packages.

Using apt-cyg

Before installing apt-cyg, you must install GnuPG and wget in Cygwin. The wget utility requires the proxy configuration to be set appropriately at /etc/wgetrc.
apt-cyg is not included among the Cygwin packages. It has to be downloaded and installed. You can display apt-cyg on your browser and then save it in a file.
git clone https://github.com/kou1okada/apt-cyg.git
cd apt-cyg
chmod +x apt-cyg
cp ./apt-cyg /usr/local/bin
Next, you need to configure a mirror and perform an update.
apt-cyg --mirror http://ftp.iij.ad.jp/pub/cygwin/
apt-cyg update
This makes it possible to download and install programs from the command line.
Examples are shown below.
apt-cyg install tar gzip # Install tar/gzip.
apt-cyg install make # Install make.
apt-cyg install cmake # Install cmake.
apt-cyg install gcc-g++ binutils # Install gcc.
apt-cyg install zlib zlib-devel # Install zlib.
apt-cyg install python python-setuptools # Install python.

FAQ

This section explains commonly encountered problems while using nlib, and their workarounds.
Overloading new/delete
nlib uses new (std::nothrow) for memory allocation. When applications and other libraries overload new or delete, operator new(size_t, const std::nothrow_t&) must also be overloaded.
For more information, see the heap_replace_malloc sample.
Overriding Functions Like malloc
nlib does not use malloc or free directly. Instead, it uses functions like nlib_malloc and nlib_free. These functions are weakly defined, so they can be overridden by user-defined functions.
You must override these functions if you want to use a user-defined heap. The following functions are required.
void* nlib_malloc(size_t size);
void nlib_free(void* ptr);
void* nlib_calloc(size_t nmemb, size_t size);
void* nlib_realloc(void* ptr, size_t size);
void* nlib_memalign(size_t alignment, size_t size); // Not defined if WIN32 or CTR.
For more information, see the heap_replace_malloc sample.
Compile Options
Platform-specific switches and other such definitions are defined in files such as cmake/common.cmake.win32 within the package. Please reference them.
Compilation results in "MACRO_XXXX not defined, compile may fail" being displayed.
You need to define MACRO_XXXX using the command-line arguments to the compiler.
Compilation results in "Please update your Visual Studio NNNN" being displayed.
You need to install the latest update for the version of Visual Studio you are using.
Should we use the latest version for our project?
That's best left up to the judgment of the project using nlib. Writing acceptance tests for future use makes it easier to make this type of decision. However, only the latest version receives support. Note this fact.
Please release a special version of the package for our project.
There are as many project requirements as there are projects. For example, in the past, inquiries like the following have been received.
  • We don't need library X, so we'd like to have a special release with that removed.
  • We'd like a release without any samples.
  • We are using the 32-bit version of Visual Studio 2012, so we'd like a release that comes with a pre-built Visual Studio 2012 solution.
To handle such requirements in a flexible manner, the projects using nlib are permitted to restructure or repackage nlib and to redistribute it, but only within the project.
Why can't text encoded using Shift-JIS be displayed with nlib_printf?
The input string for nlib_printf must be encoded using UTF-8.
There is a bug in a version released half a year ago. Will you please release a patch that fixes it?
Bug fixes are applied to the latest version. You need to use a newer release that incorporates the fix.
Will you please make it follow our library or framework's approach?
If your approach is publicly available, and is supported by all platforms (Windows, Linux, FreeBSD, OS X, Cafe, and CTR), this might be possible. Please get in touch.

Revision History

2015-09-08

  • Started operation check in FreeBSD 10.2. FreeBSD 10.1 is no longer officially supported.
  • Added the DynamicAlignedStorage class to the misc library.
  • Added the LockFreeUnitHeap class to the misc library.
  • Added the LockFreePipe class to the misc library.
  • Added the LockFreeStack class to the misc library.
  • Added the LockFreeQueue class to the misc library.
  • Added the LockFreeInit() function to the misc library.
  • Added the LockFreePriorityQueue class to the misc library.
  • Added the LockFreeBroadcastQueue class to the misc library.
  • An alignment that is larger than 4096 can now be specified in nmalloc_aligned().
  • Fixed an issue where NULL was returned when a too large size was specified in nmalloc() or nrealloc().
  • The computation speed in the Crc32 class has been now improved.
  • Fixed a bug where a destructor was not executed properly when a message queue with a priority set was used in a lock free manner.
  • Deleted deprecated files and functions.

2015-07-28

2015-06-26

  • Added support for Fedora 22. Added support for builds that use gcc 5.1.
  • Started operation check in FreeBSD 10.1. FreeBSD 10.0 is no longer officially supported.
  • Updated the version of LZ4 bundled with the oss libraries to 1.7.0.
  • Added nn::nlib::oss::Lz4InputStream and nn::nlib::oss::Lz4OutputStream to the oss library.
  • Added the lz4_nlib command-line compression tool that uses LZ4.
  • Added the HandleMaker class to the misc library. This class supports implementations that allow a handle (32-bit integer value) to be associated with some objects.
  • Added the nlib_spinlock_init(), nlib_spinlock_lock(), nlib_spinlock_trylock(), and nlib_spinlock_unlock() functions to the misc library.
  • Added the nlib_thread_exit() and nlib_thread_exit_cpp() functions to the misc library.
  • Added the nlib_thread_attr_setstack() and nlib_thread_attr_getstack() functions to the misc library. Along with this addition, deleted the NLIB_THREAD_ATTR_KEY_STACKADDR macro.
  • Added the nlib_error_string() function to the misc library.
  • Changed the specification of the threading::ThreadSettings class in the misc library to wrap nlib_thread_attr. Related code needs to be modified since the source code level compatibility is no longer provided.
  • The operations of nmalloc can now be customized with the NLIB_NMALLOC_DISABLE_TLSCACHE, NLIB_NMALLOC_HEAPSIZE, and NLIB_NMALLOC_MODE environment variables.
  • Fixed an issue where nmalloc_aligned() might not allocate memory with a specified alignment.
  • Improved the hash quality of nmalloc_heaphash().
  • Fixed an issue where the sample XML file for the testing library was not output in the 2015-05-26 version.
  • nlib_strerror() is now deprecated.
  • threading::CondVar::Init() is now deprecated.
  • Made various improvements and fixed bugs in platform code.
  • Made other minor changes and fixes.

2015-05-26

  • Updated the version of LZ4 bundled with the OSS libraries to 1.6.0.
  • Added the following macros to the testing library: ASSERT_STRCASEEQ, ASSERT_STRCASENE, EXPECT_STRCASEEQ, and EXPECT_STRCASENE.
  • The testing library now loads several environment variables supported by Google Test.
  • Added the nlib_disk_freespace function to the misc library.
  • Added the nlib_fd_readv and nlib_fd_writev functions to the misc library.
  • Added the nlib_fd_preadv and nlib_fd_pwritev functions to the misc library.
  • Added the OutputStream::WriteGather member function to the OutputStream class in the misc library.
  • Added the Nflags::GetBoolFromEnv, Nflags::GetInt32FromEnv, Nflags::GetInt64FromEnv, Nflags::GetDoubleFromEnv, and Nflags::GetStringFromEnv member functions to the Nflags class in the misc library.
  • Started operation check in Ubuntu 15.04. Ubuntu 14.10 is no longer officially supported.
  • There is no longer an HTML Help version of the reference manual. This version is no longer necessary now that the functionality of the Doxygen search box has been improved.
  • Deleted deprecated files and functions.
  • StringInputStream is now deprecated.
  • Made other minor changes and fixes.

2015-04-21

  • Added support for CentOS 7.1. Added support for builds that use gcc.
  • Made improvements to the layout of the Reference Manual (English version).
  • Made other minor changes and fixes.

2015-03-31

  • Made improvements to the layout of the Reference Manual.
  • Added the 64-bit static library for Visual Studio 2012 to the package.
  • Added the 64-bit DLL for Visual Studio 2012 and 2013 to the package.
  • Divided the oss library files for each OSS. The reason being that the OSS binaries cannot be excluded during the dead-code stripping process that occurs when linking DLLs and shared libraries (but not static libraries).
  • Added a suffix of '_d' to the basename for library files in the debug build. This change prevents DLLs and shared libraries from mistakenly dynamic linking to these files.
    • The CAFE and CTR library files are exceptions, however.
  • Made revisions to put the debug and release builds of the libraries in the same directory. This change is possible because the filenames for the different builds are no longer the same.
    • The CAFE and CTR library files are exceptions, however.
  • A string representing the version and build settings will be added to the base names of library files for Visual Studio builds. In conjunction with this, files will all be stored in the msvc directory, instead of in directories created for each version and build setting.
    • For example, library files for the 32-bit, static library version of Visual Studio 2012 will be named nx_misc-vc120.lib and nx_misc-vc120_d.lib.
  • The command-line tool can now access shared libraries via relative paths in an environment in which RPATH can be configured. As a result of these changes the LD_LIBRARY_PATH setting is no longer needed.
  • The DateTime class now supports reads and writes in asctime format.
  • Made other minor changes and fixes.

2015-02-23

  • Added an English version of the reference manual (this document).
  • Added support for OS X. Added support for builds that use Xcode (clang).
  • Added libraries compiled for CAFE and CTR to the PC version of the package.
  • Added nlib_mlock and nlib_munlock.
  • msgpack::MpObject now supports reading and writing by C++11 standard library array, unordered_map, and tuple objects.
  • Added the NLIB_VIS_HIDDEN and NLIB_VIS_PUBLIC macros.
  • Changed builds in Visual Studio 2010 and later versions to use the DLL version of the C runtime.
    • Changed the /MT and /MTd options to /MD and /MDd, respectively.
  • Added the -fvisiblity=hidden and -fvisibility-inlines-hidden options to the compiler options for gcc and clang libraries in non-Cygwin environments. This reduces the number of symbols the library makes public.
    • Generally, the load time of shared libraries has improved.
  • When using CMake, you must use version 3.1.0 or later.
  • nn::nlib::threading::CondVarFallback is now deprecated. It will be deleted in a future release.
  • Made other minor changes and fixes.

2015-01-14

2014-12-10

  • Added support for the 64-bit version of FreeBSD 10.0. Added support for builds using clang and libc++.
  • Added nlib_clz, nlib_clz64, nlib_ctz, and nlib_ctz64.
  • Added nlib_popcnt, nlib_popcnt64, and nlib_popcnt16.
  • Added nn::nlib::simd::I128::AlignR.
  • Improved the performance of code written for SSE4.1.
  • Revised the CAFE implementation of nlib_ticktime to use OSGetSystemTime instead of OSGetTick.
  • Improved finalization (unloading) of the heap library (for the case of DLL/shared libraries).
  • Made other minor changes and fixes.

2014-11-18

  • Improved the testing library.
    • Added support for the options - --gtest_filter, --gtest_shuffle, --gtest_random_seed, and --gtest_print_time.
    • Added support for the - --help option.
  • Fixed a bug with the heap library that caused metadata to become corrupted in some situations.
  • Made other minor changes and fixes.

2014-11-07

  • Added Readme-ja.html.
  • Made other minor changes and fixes.

2014-11-04

  • Improved the performance of the heap library.
  • Added nmalloc_get_settings.
    • When using nmalloc, use nmalloc_get_settings instead of configuring g_NMallocSettings.
  • Added nlib_gen_random.
  • Improved the performance of nn::nlib::succinct::AhoCorasickBuilder.
  • Improved the performance of nn::nlib::threading::ThreadPool.
  • Updated the version of SQLite bundled with the OSS libraries to 3.8.7.
  • Fixed a bug that sometimes prevented the PC version of nrealloc from allocating physical memory appropriately.
  • Fixed a bug that prevented occasional error detection in the SIMD-optimized version of nlib_strcplen_ex.
  • Made other minor changes and fixes.

2014-10-01

  • It is now possible to specify the destructor in the CTR version of nlib_tls_alloc.
    • Starting with this release, CTR-SDK 10.0 or later is required.
  • Added the gameheap sample to the heap library.
    • This sample builds and uses heaps in separate per-object, per-module, and per-thread memory regions.
    • The use of multiple different heaps may make it possible to reduce the risk of memory-related bugs.
    • Each heap operates in a thread-safe manner and keeps fragmentation in check.
    • It uses the heap library to implement an interface that is typical of heaps used in game development.
  • Fixed a bug that caused a leak in the thread-cache region of the heap library's nmalloc function.
    • This bug occurred in environments in which nlib_tls_getvalue returns NULL when a thread ends.
  • The nmalloc_thread, nmalloc_realloc, nmalloc_malicious, and speeddemo samples were deleted, and their contents were integrated into thenmalloc_simple sample.
  • When using the text-based XML parser of the exi library, revised the behavior that caused strings in text nodes to be invalid prior to allocator deletion.
    • This behavior has been revised to match the behavior of the binary XML parser, so that strings for text nodes are valid until the allocator is deleted.
  • Deleted XmlStreamEventLogger from the exi library.
  • Made other minor changes and fixes.

2014-09-03

  • Improved checks for macro definitions in the PC environment. This mitigates problems that occur when compiling using proprietary makefiles or project files.
    • Displays a warning message if the following macros are not defined during compilation in Windows.
      • WIN32_LEAN_AND_MEAN, NOMINMAX, _USE_MATH_DEFINES
    • Displays a warning message if the following macros are not defined during compilation in Windows, Cygwin, or Linux.
      • __STDC_LIMIT_MACROS, __STDC_CONSTANT_MACROS, __STDC_FORMAT_MACROS
  • Changed the typedef for nlib_sock from SOCKET type to int type in Windows.
  • Rebuilt NEON code for the simd library using the actual cross compiler for the ARM architecture (gcc 4.9).
    • Operational tests currently use the NEONvsSSE_6.h file distributed by Intel.
  • Added AArch64 code to the code for the simd library.
  • Improved the instructions for setting up nlib.
  • Added an FAQ.
  • Deprecated several functions in the nn::nlib::unicode namespace. These will be deleted in a future release.
    • nn::nlib::unicode::ToUtf32, nn::nlib::unicode::ToUtf16, and nn::nlib::unicode::ToUtf8
    • nn::nlib::unicode::Utf32ToUtf8, nn::nlib::unicode::Utf16ToUtf8, and nn::nlib::unicode::WideToUtf8
    • nn::nlib::unicode::Utf8ToUtf32, nn::nlib::unicode::Utf8ToUtf16, and nn::nlib::unicode::Utf8ToWide
  • Deleted several header files, functions, and the like.
    • Deleted nn/nlib/threading/SimpleRingBuffer.h from the misc library.
    • Deleted nn/nlib/threading/Interlocked.h from the misc library. In the future, use a C-linkage function like nlib_atomic32_increment directly.
    • Deleted nn/nlib/threading/CallOnce.h from the misc library.
      • Deleted functions like nn::nlib::threading::CallOnce and nn::nlib::threading::TryCallOnce. In the future, use C-linkage functions like nlib_once and nlib_tryonce directly.
    • Deleted the NLIB_PRINTF, NLIB_VSNPRINTF, and NLIB_VSNWPRINTF macros.
    • Deleted the NLIB_ATTRIBUTE_ALIGN macro. In the future, these macros will be integrated into the NLIB_ALIGNAS macro.
    • Deleted nn::nlib::StrEqual and nn::nlib::StrNEqual. Use a function like strcmp or nn::nlib::StrCmp directly.
  • Revised several samples.
  • Updated NLIB_MEMORY_ORDER_RELEASE, NLIB_MEMORY_ORDER_ACQUIRE, and NLIB_MEMORY_ORDER_ACQ_REL.
    • atomic_thread_fence is used if C++11 is available.
    • Added optimization barriers for functions like _WriteBarrier on the Win32 version. This resolves some problems caused by optimization.
    • In the CTR version, run a combination of DataSynchronizationBarrier and optimization barriers.
  • Made other minor changes and fixes.

2014-08-04

  • Made the simd library support paired singles on CAFE. Calculations that use single-precision floating-point numbers now use paired singles.
    • Note that this method is significantly slower than the Cafe SDK functions for purposes such as matrix calculation, because its interface returns 128-bit data such as four-dimensional vectors as values.
    • On SSE4.1/NEON, this method is optimal because it stores and returns 128-bit data in 128-bit registers like the XMM register.
  • Added classes and functions to the simd library.
    • Added the Frustum class.
    • Added methods for determining whether a frustrum intersects or contains various types of geometry.
  • In the CTR version, made changes so that nlib_init_ctrsample allocates the buffer for RomFs dynamically.
  • Adjusted the initial size of the buffer allocated for XML attributes by the tokenizer of the text-based XML parser of the exi library.
  • In the Win32 environment, made changes so that the UNICODE and _UNICODE macros are no longer defined. nlib does not depend on the existence of these macro definitions.
  • In the Win32 environment, made changes to check the value of the WINVER macro.
  • Added an introduction to the Reference Manual about methods of automating setup of the build environment.
    • Added a description of the chocolately package-management tool for Windows. Using chocolately enables you to implement a degree of automation for the build environment for Windows program development.
    • Added a description of the about the apt-cyg package-management tool for Cygwin. Using apt-cyg enables you to implement a degree of automation for the build environment for Cygwin program development.
  • Improved the explanation of how to set up the environment for using nlib with Windows and Cygwin.
  • Fixed a bug that prevented versions using the nlib_strcplen and nlib_strcplen_ex SIMD instructions from operating correctly when strings were terminated immediately before page boundaries (4096-byte boundaries).
  • Made other minor changes and fixes.
Contains Past Revision Histories.