nlib
nlib Main Page

This manual is a collection of libraries that support Windows, Linux, FreeBSD, OS X, Cygwin, and Nintendo's development environments.

About nlib

nlib is being developed to provide a group of user-space libraries that enable you to program for Windows, Linux, FreeBSD OS X, Cygwin, and Nintendo's development environments using the same interface. Its development follows a software-centric approach, adopts appropriate and pragmatic coding rules (nlib Coding Rules), and implements an interface that is independent from the individual development environments. This focus makes it possible to develop in parallel for multiple development environments while maintaining high quality, performance, and reliability.

Features Implemented by nlib

The main features implemented are listed below.
  • Reading and writing XML and JSON
  • Highly-efficient memory allocators
  • Test framework compatible with Google Test
  • Wrapper libraries that allows codes to be developed that supports both SSE4.2 and NEON. Includes support for various calculations for 3x4 matrices, and three- and four-dimensional vectors.
  • Succinct data structure libraries for features like keyword searching
  • UTF-8 reading, writing, normalization, and conversion from and to UTF-16 and UTF-32
  • Threads, mutexes, condition variables, TLS, barriers, read-write locks, and atomic variables
  • Getting and freeing system memory, getting the time, and reading and writing files
  • Basic lock-free data structure and algorithms
  • Functions and macro definitions for base C linkages used to provide multi-platform and multi-compiler support

nlib Operating Environment

Currently, nlib has been verified for compilation and operation in the following environments.
Platform Library types Remarks
Windows Visual Studio 2015 64bit static library, DLL Update 3
32bit static library
Visual Studio 2013 64bit static library, DLL Update 5, Express Edition is available
32bit static library
Visual Studio 2012 64bit static library, DLL Update 5, Express Edition is available
32bit static library
Cygwin 64bit static library gcc 5.4.0, newlib, libstdc++
Bash on Ubuntu on Windows 64bit shared library gcc 4.8.4, clang 3.4, glibc, libstdc++
Linux Ubuntu 16.04 LTS 64bit shared library gcc 5.3.1, clang 3.8, glibc, libstdc++, https://atlas.hashicorp.com/ubuntu/
Ubuntu 14.04 LTS 64bit gcc 4.8.4, clang 3.4, glibc, libstdc++, https://atlas.hashicorp.com/ubuntu/
32bit
CentOS 7 (v1608.01) 64bit gcc 4.8.5, glibc, libstdc++, https://atlas.hashicorp.com/centos/
Fedora 24 64bit gcc 6.1.1, clang 3.8, glibc, libstdc++. https://atlas.hashicorp.com/fedora/
Alpine 3.4.0 64bit gcc 5.3.0, musl libc, libstdc++, https://atlas.hashicorp.com/maier
FreeBSD FreeBSD 11.0 64bit shared library clang 3.8, BSD libc, libc++, https://atlas.hashicorp.com/freebsd
OS X OS X 10.11 (El Capitan) 64bit shared library Xcode 8.0
CTR CTR-SDK 11.5 32bit static library CTR_SDK-11_5_1-20160622-ja.zip
armcc_4_1_nintendo-b1454-20150227.zip
Cafe CafeSDK 2.13.01 32bit static library cafe_sdk-2_13_01-20150728.zip
MULTI-5_3_26-20150918.zip
Depending on the compiler version and other factors, compilation uses C++11 features as appropriate (C++11 Features Used by nlib).

nlib Support Policy

Please contact the site where the package was released for instructions about how to get support. Also read the documentation, including the FAQ. This document and the code samples explain the current implementation. Future development plans and release dates will not be documented or made public. Support and bug fixes will be made in the latest version at that time. Special versions will not be provided for specific projects, and no special support will be provided. This rule is essential to keep the size of the development team in check.
Unannounced changes that break compatibility may occur if there is no way around it. Note this fact.
Before use, it is probably a good idea to perform some acceptance tests to determine performance criteria including whether the package works as expected, is sufficiently fast, and has memory consumption within the acceptable limits. Creating acceptance tests up front also helps to avoid trouble when upgrading nlib. It is a good idea to use the testing library to create these types of acceptance tests.
The bug may be quickly fixed if you describe the code reproducing the bug as a test code in the nn::nlib::testing format and include it in your bug report.

nlib Package Description

Note that the included libraries and supported platforms vary depending on the package.

Directory Structure

The package is structured as follows.
    ├── bin
    │   └── cmake                 # Command-line tools
    ├── cmake                      # Common Cmake scripts
    ├── documents-en
    │   └── API                   # Reference Manual
    │       ├── html              # HTML version of the Reference Manual
    │       └── searchdata.xml    # Search index
    ├── include                    # Include files (in `system/include` for Cafe)
    │   ├── nn
    │   │   └── nlib
    │   │       ├── exi
    │   │       ├── heap
    │   │       ├── msgpack
    │   │       │   └── jsonrpc
    │   │       ├── oss
    │   │       ├── simd
    │   │       ├── succinct
    │   │       ├── testing
    │   │       ├── threading
    │   │       └── unicode
    │   ├── lz4.h
    │   ├── lz4hc.h
    │   ├── lz4frame.h
    │   ├── sqlite3.h
    │   └── sqlite3ext.h
    ├── lib                        # Platform-specific library files (`system/lib` for Cafe, `libraries` for CTR)
    ├── samples                    # Source code for samples
    │  ├── cmake
    │  └── sources
    │      ├── exi
    │      ├── heap
    │      ├── misc
    │      ├── msgpack
    │      ├── oss
    │      ├── succinct
    │      └── testing
    └── setup
        ├── nlib.natvis
        ├── setup-bash-ubuntu-windows-tools.sh
        ├── setup-cygwin-tools.sh
        └── setup-msvc-libraries.sh
        └── setup-vagrant-plugins.sh

Platform-Specific Descriptions

Descriptions of Each Library

\par <tt>libn</tt> Library (including <tt>libn.a, libn.so, n-vc120amd64.lib, and n-vc120dllamd64.dll</tt>, etc.)
This library includes functions that have C linkages for suppressing differences among various platforms.
\par
Specifically, the following functions are provided:
<ul>
<li> Functions, including <tt>nlib_epochtime()</tt>, that obtain the current time.
<li> The <tt>nlib_gen_random()</tt> function that obtains random values.
<li> Allocation and deallocation of (virtual) memory from the operating system
<li> Functions and macros supporting pthread
 <ul>
 <li> Threads (\ref nlib_thread) and functions that start a thread or merge threads.
 <li> Mutexes (<tt>\ref nlib_mutex</tt>)
 <li> Conditional variables (<tt>\ref nlib_cond</tt>)
 <li> TLS (<tt>\ref nlib_tls</tt>)
 <li> Barriers (<tt>\ref nlib_barrier</tt>)
 <li> One-time execution (<tt>nlib_once()</tt>, <tt>nlib_tryonce()</tt>)
 <li> Read-write lock (<tt>\ref nlib_rwlock</tt>)
 <li> Conditional variables for read-write lock (<tt>\ref nlib_condrwlock</tt>)
 </ul>
<li> Message queues supporting lock-free behavior (<tt>\ref nlib_mq</tt>)
<li> Functions, including <tt>nlib_atomic_compare_exchange32()</tt>, that atomically rewrite memory.
<li> Functions, including <tt>nlib_debug_break()</tt> and <tt>nlib_debug_backtrace()</tt>, that access the debugger.
<li> The <tt>nlib_getenv()</tt> function that obtains environment variable values.
<li> File handles and file I/O functionality, including <tt>nlib_fd_open()</tt>.
<li> Functions relating to string searches, accelerated with SIMD.
<li> String encoding conversion between UTF-8 and UTF-16 and between UTF-8 and UTF-32, in both directions
<li> Functions, including <tt>nlib_strto_int32()</tt>, that wrap standard C functions, including <tt>strtol()</tt> and <tt>strtoll()</tt>, in a way they are not wrongly used.
<li> Functions, including <tt>nlib_isspace()</tt>, that implement standard C functions defined by <tt>ctype.h</tt> in a way they are not affected by the locale.
<li> Functions, including <tt>nlib_snprintf()</tt>, that are adjusted to suppress differences among the environment-specific behaviors of <tt>snprintf()</tt>, <tt>snwprintf()</tt>, and other similar functions to prevent them from being wrongly used.
<li> Fast conversion of endianness (including <tt>nlib_swapendian_32()</tt>)
<li> Fast CRC32 and CRC32C calculations (<tt>nlib_crc32()</tt> and <tt>nlib_crc32c()</tt>)
<li> Fast pop count calculations (including <tt>nlib_popcnt64()</tt>)
<li> Macros that absorb differences among the various attributes for each compiler
</ul>

\par misc Library (libnx_misc.a, libnx_misc.so, nx_misc-vc120amd64.lib, nx_misc-vc120dllamd64.dll, etc.)
This library is a class library that includes relatively small classes.
It implements many features that are believed to be commonly used in programs.
\par
Specifically, it includes the following.
<ul>
<li> Base stream classes that serve as a foundation for <tt>InputStream</tt> and <tt>OutputStream</tt>
 <ul>
 <li> Reading and writing data to memory (<tt>\ref MemoryInputStream</tt> and <tt>\ref MemoryOutputStream</tt>)
 <li> Reading and writing files (<tt>\ref FileInputStream</tt> and <tt>\ref FileOutputStream</tt>)
 <li> Reading and writing Base64-encoded data (<tt>\ref Base64InputStream</tt> and <tt>\ref Base64OutputStream</tt>)
 <li> Reading and writing <tt>zlib</tt>-compressed data (<tt>\ref ZlibInputStream</tt> and <tt>\ref ZlibOutputStream</tt>)
 </ul>
<li> Classes that read and write UTF-8 strings from streams while validating (<tt>\ref TextReader</tt> and <tt>\ref TextWriter</tt>)
<li> Classes that read and write binary data from streams (<tt>\ref BinaryReader</tt> and <tt>\ref BinaryWriter</tt>)
<li> Classes for handling Unicode
 <ul>
 <li> Classes for Unicode string normalization (NFC, NFD, NFKC, NFKD) (<tt>\ref unicode::UnicodeNormalizer</tt>)
 <li> Tables for RFC 3454 (<tt>\ref unicode::StringPrep</tt>)
 </ul>
<li> Classes and functions for handling threads
 <ul>
 <li> A wrapper class of types and functions defined in the <tt>libn</tt> library.
 <li> A class corresponding to <tt>std::thread</tt> that runs without C++03 or C++ exceptions (<tt>\ref threading::Thread</tt>)
 <li> Classes and functions corresponding to <tt>std::future</tt> and <tt>std::async</tt> that run without C++03 or C++ exceptions (<tt>\ref threading::Future</tt> and <tt>\ref threading::Async</tt>)
 <li> Thread pools (<tt>\ref threading::ThreadPool</tt>)
 </ul>
<li> Classes and functions for handling date and time information
 <ul>
 <li> Classes for adding and subtracting dates and times, in addition to parsing and creating RFC 2822 or W3C DTF date strings (<tt>\ref DateTime</tt> and <tt>\ref TimeSpan</tt>)
 </ul>
<li> Classes and functions for handling URIs
 <ul>
 <li> URI parsers (<tt>\ref Uri</tt>)
 <li> Classes for processing RFC 6570 URI templates (<tt>\ref UriTemplate</tt>)
 <li> A class for standardizing file paths, which are often represented by different strings depending on the environment, into URI strings (<tt>\ref NativePathMapper</tt>).
 </ul>
<li> A class for handling asynchronous file access (<tt>\ref threading::AsyncFileIo</tt>)
<li> Basic lock-free data structure and the algorithm
 <ul>
 <li> A function, including <tt>LockFreeInit()</tt>, for initializing lock-free objects.
 <li> A lock-free pipe (<tt>LockFreePipe</tt>) that can be used when one sender thread is associated with only one receiver thread.
 <li> A lock-free stack (<tt>LockFreeStack</tt>)
 <li> A lock-free queue (<tt>LockFreeQueue</tt>)
 <li> A lock-free queue with a priority set (<tt>LockFreePriorityQueue</tt>)
 <li> A lock-free broadcast queue (<tt>LockFreeBroadcastQueue</tt>)
 </ul>
<li> The different types of gadget classes
 <ul>
 <li> A substitute class for <tt>std::unique_ptr</tt> (<tt>\ref UniquePtr</tt>). In C++11, this class is defined as <tt>std::unique_ptr</tt> using a <tt>typedef</tt>.
 <li> %Singleton classes (<tt>\ref Singleton</tt> and <tt>\ref SimpleSingleton</tt>)
 <li> An <tt>std::vector</tt>-like class that does not require freeing memory and copying elements upon expansion (<tt>\ref Nlist</tt>)
 <li> A class for parsing command-line strings (<tt>\ref Nflags</tt>)
 <li> A class for supporting handle creation (<tt>\ref HandleMaker</tt>)
 </ul>
</ul>
The namespaces include <tt>nn::nlib</tt>, <tt>nn::nlib::unicode</tt>, and <tt>nn::nlib::threading</tt>.

\par exi Library (libnx_exi.a, libnx_exi.so, nx_exi-vc120amd64.lib, nx_exi-vc120dllamd64.dll, etc.)
A library used to read and write binary XML. It can also read and write text-based XML.
It supports XML namespaces and includes a simple, easily programmable pull parsing API. \n
The namespace is <tt>nn::nlib::exi</tt>.

\par heap Library (libnx_heap.a, libnx_heap.so, nx_heap.lib, nx_heap-vc120dllamd64.dll, etc.)
This library is a heap library for which memory allocation and deallocation is optimized with multithreaded operations. The <tt>nmalloc</tt> and <tt>nfree</tt> functions (and the <tt>heap::CachedHeap</tt> and <tt>heap::CentralHeap</tt> classes used to implement them) keep performance degradation in check by controlling fragmentation through the use of multi-level free lists.
By maintaining per-thread caches for small blocks of memory, memory can be allocated and freed without locking, which helps to limit the degradation of multithreaded performance.
Pointers within the free lists are checked for both mangling and their allocation status. This implementation makes it more difficult to use them in an invalid way by overwriting memory externally. \n
The namespaces are <tt>nn::nlib::heap</tt> and the global namespace (C linkage functions).

\par succinct Library (libnx_succinct.a, libnx_succinct.so, nx_succinct-vc120amd64.lib, nx_succinct-vc120dllamd64.dll, etc.)
A library used to handle succinct data structures.
You can think of a succinct data structure as one whose content can be searched while it is still compressed. \n
The <tt>succinct</tt> library includes the following.
<ul>
<li> Implementation of succinct vector classes
<li> A class that implements balanced parentheses
<li> A class that implements LOUDS (level-order unary degree sequences)
<li> Trie implementation
<li> A class for creating high-compression indices and searching strings using the Aho–Corasick algorithm
</ul>
The namespace is <tt>nn::nlib::succinct</tt>.

\par msgpack Library (libnx_msgpack.a, libnx_msgpack.so, nx_msgpack-vc120amd64.lib, nx_msgpack-vc120dllamd64.dll, etc.)
A library for reading and writing <a href="http://msgpack.org/">MessagePack</a> and JSON.
It also includes a class for parsing CSV. \n
The <tt>msgpack</tt> library includes the following.
<ul>
<li> Classes for reading and writing JSON from streams
<li> Classes for reading and writing MessagePack from streams
<li> A class for high-speed in-memory MessagePack parsing
<li> A class for reading CSV
<li> Classes for storing JSON, MessagePack, and CSV data after it has been read
<li> Implementation of JSON-RPC
</ul>
The namespace is <tt>nn::nlib::msgpack</tt>.

\par testing Library (libnx_testing.a, libnx_testing.so, nx_testing-vc120amd64.lib, nx_testing-vc120dllamd64.dll, etc.)
A library for writing highly portable test code.
It is a testing framework that is somewhat source-code compatible with tests written for <a href="https://code.google.com/p/googletest/">Google Test</a>. \n
The namespace is <tt>nn::nlib::testing</tt>. (It is aliased to the <tt>testing</tt> namespace for the sake of Google Test compatibility.) It depends on the <tt>misc</tt> library.

\par oss Library (libnx_oss_bsdiff.a, libnx_oss_sqlite3.a, libnx_oss_lz4.a, etc)
Contains code for open-source software (OSS), code derived from OSS, and code for accessing OSS.
To use it, an OSS license must be processed. \n
The following libraries are currently included. They are built as separate library files.
<ul>
<li> A library conversion of <a href="http://www.daemonology.net/bsdiff/">bsdiff</a> that has been modified to use <tt>zlib</tt> instead of <tt>bzlib2</tt>, and the corresponding command-line programs
<li> A version of <a href="https://www.sqlite.org/">SQLite3</a> that has been modified to run using the interfaces provided by <tt>nlib</tt>, and a corresponding command-line shell
<li> <a href="https://github.com/Cyan4973/lz4">LZ4</a> is approximately 20 times faster at data compression than <tt>zlib</tt>, and approximately five times faster at decompression.
</ul>
\par
The namespaces are <tt>nn::nlib::oss</tt> and the global namespace (C linkage functions).

\par simd Library (Mostly Inline Functions Included in the misc Library)
SIMD instructions can significantly increase the speed of programs.
The <tt>simd</tt> library makes it possible to write code that supports both SSE4.2 and NEON. \n
It supports integer and single-precision floating-point number calculations that enable your code to compute at high speeds. \n
Single-precision floating-point number calculation can be used even in environments that do not support SSE or NEON. \n
On Cafe, this is optimized to use paired singles.
\par
In other words, the same code can support the following environments.
<ul>
<li> On x86 processors, the code is compiled with speed boosts through the use of SSE 4.2.
<li> On processors with NEON support, the code is compiled with speed boosts through the use of NEON.
<li> On Cafe, the code is compiled with speed boosts through the use of paired singles. (Floating-point operations alone are quite a bit slower than the Cafe-SDK for operations like matrix calculations, because the interface returns values for data like four-dimensional vectors.)
<li> On CTR, generic code is compiled (floating-point operations only).
</ul>
The namespace is <tt>nn::nlib::simd</tt>.

Dependency between nlib libraries.

The dependency between nlib libraries is as shown in the following figure.

dot_inline_dotgraph_32.png

C++11 Features Used by nlib

nlib is compiled using C++11 features for some combinations of compilers and versions. The table below summarizes the C++11 features, compatibility macros that can be used with nlib, and the compilers that compile using C++11. Note that for clang, the determination is made based on __has_feature and __has_include macros, and not the version.
C++11 features are not used in any other environment.
C++11 Feature Identification Macro Macros and Typedefs for C++03 Compatibility Visual Studio gcc
Rvalue references NLIB_CXX11_RVALUE_REFERENCES None 2010 4.3
Variadic templates NLIB_CXX11_VARIADIC_TEMPLATES None 2013 4.7
Static assertions NLIB_CXX11_STATIC_ASSERTIONS NLIB_STATIC_ASSERT() 2010 4.3
Null pointer constant NLIB_CXX11_NULL_POINTER_CONSTANT nn::nlib::nullptr, nn::nlib::nullptr_t 2010 4.6
Deleted functions NLIB_CXX11_DEFAULTED_AND_DELETED_FUNCTIONS NLIB_DISALLOW_COPY_AND_ASSIGN() 2015 4.4
Explicit conversion operators NLIB_CXX11_EXPLICIT_CONVERSION_OPERATORS NLIB_SAFE_BOOL() 2013 4.6
Alias templates NLIB_CXX11_TEMPLATE_ALIAS None 2013 4.7
constexpr NLIB_CXX11_CONSTEXPR NLIB_CEXPR 2015 4.7
char16_t, char32_t NLIB_CXX11_NEW_CHARACTER_TYPES nlib_utf16_t, nlib_utf32_t 2015 4.4
alignas NLIB_CXX11_ALIGNMENT_SUPPORT NLIB_ALIGNAS() 2015 4.8
Explicit virtual overrides NLIB_CXX11_EXPLICIT_VIRTUAL_OVERRIDES NLIB_OVERRIDE 2012 4.7
final NLIB_CXX11_EXPLICIT_VIRTUAL_OVERRIDES NLIB_FINAL 2012 4.7
noexcept NLIB_CXX11_NOEXCEPT NLIB_NOEXCEPT 2015 4.6
Range-based for NLIB_CXX11_RANGE_BASED_FOR None 2012 4.6
lambda NLIB_CXX11_LAMBDAS None 2012 4.7
Delegating constructors NLIB_CXX11_DELEGATING_CONSTRUCTORS None 2013 4.7
std::unique_ptr NLIB_CXX11_UNIQUEPTR nn::nlib::UniquePtr 2013 4.7
#include <random> NLIB_CXX11_STDLIB_RANDOM None 2013 4.8
#include <chrono> NLIB_CXX11_STDLIB_CHRONO Conversion to and from nn::nlib::DateTime 2012 4.7
#include <atomic> NLIB_CXX11_STDLIB_ATOMIC NLIB_MEMORY_ORDER_SEQ_CST and others 2012 4.8
#include <array> NLIB_CXX11_STDLIB_ARRAY None 2012 4.5
#include <unordered_map>
#include <unordered_set>
NLIB_CXX11_STDLIB_UNORDERED None 2012 4.5
#include <tuple> NLIB_CXX11_STDLIB_TUPLE None 2012 4.5

nlib Coding Rules

nlib Basic Coding Rules

The nlib coding rules are a slight modification and relaxation of the latest version of the Google C++ Style Guide (and a slightly outdated Japanese translation).
The reasons for using the Google C++ Style Guide are listed below.
  • It is easy to access documentation about the coding rules. Translations are available (albeit unofficial ones).
  • Similar to the general coding rules for the gaming industry, C++ exceptions and RTTI use are prohibited.
  • It has been updated continuously in the past.
  • Changes made to the coding rules with each update are reasonable, and there haven't been any sudden, drastic changes.
  • We can expect it to be maintained even five years from now.
The changes and relaxations adopted for nlib are shown below.
Coding rule Google C++ Style Guide nlib
Line length 80 100
Indentation 2 4
Parameter order Inputs followed by outputs Outputs followed by inputs
Default argument Prohibited in principle Allowed for use except for the cases described below:
  • In the case of a constructor. Issues may arise if the default value allows an argument not to be explicitly specified until only one argument remains unhandled.
  • In the case of a virtual function. Issues may arise if the overridden function has a default argument different from that of the original function.
  • In the case where the argument can be used as a function pointer. Confusion may be induced in the coding.
File name file_name.cc FileName.cpp(file_name.cpp in the sample)
#include guard Also append #pragma once (for accelerating compilation in Visual Studio)
Use of boost Not to be used by libraries (may be used for other purposes such as testing)
Accessor function name get_xyz() / set_xyz() GetXyz() / SetXyz()
Argument passed by reference Non-const references are prohibited In nlib, non-cost references are allowed as exceptions in the following cases.
  • When specializing std::swap
  • Code that imitates move semantics in C++03
  • Class with simple features. For example, a class having a feature that only maintains a lock, such as the ScopedLock class.
  • When passing nn::nlib::UniquePtr as an argument (Returning a value would be inefficient, and it would be strange to pass a pointer to UniquePtr.)
Use of C++11 Ensure that headers and source code can be properly compiled also by using C++03 compilers.
  • This doesn't mean you can't use C++11. Features such as move constructors, range-based for statements, explicit virtual overrides, and explicit conversion operators should all be supported proactively.
  • If possible, the same source code is compiled using macros and other preprocessor directives. If not possible, implement similar features that can be used from C++03.
nullptr Use if C++11 In principle, don't use nullptr_t or nullptr as long as C++03 is used.
  • The nullptr idiom for implementing nullptr in C++03 results in errors if it is defined in multiple libraries, so it cannot be defined in the global namespace.
  • With nlib, it is defined as nn::nlib::nullptr_t and nn::nlib::nullptr for C++03.
  • Avoiding the use of misleading overloads is a workaround for most cases that would require nullptr.
constexpr Use if C++11 For the time being, use of constexpr should be avoided except in obvious cases.
  • Unfortunately, code that uses constexpr proactively has a drawback in terms of portability, because differences among compiler behavior can result in many compilation errors.
  • "Obvious cases" refers to null constant objects or cases that one would expect to compile without issues on any compatible compiler that supports constexpr (for example, simple constructors that merely set constants).
Comment
  • Considering translation costs, do not use non-ASCII (non-English) characters in comments for public headers (This rule does not apply if the content should be never translated).
  • If you want to write detailed comments, include them in either source code files or the Reference Manual.
  • Doxygen comments must not be written in public headers. This causes recompilation due to changes in the documentation, and increases the translation burden. Files such as manual_ja.h and manual_en.h should be prepared for each library and comments should be written into those files.
  • In the reference manuals, never place newline characters in the middle of a sentence. This rule is meant to support the nature of the translation-aid software used for localization. It takes precedence over the previously mentioned limit of 100 characters per line.

Checking and Correcting Coding Style

The coding style can be checked easily by using cpplint.
The warning criteria have been slightly relaxed, as follows.
  • Specify --linelength=100 to allow up to 100 characters per line.
  • Warnings due to cpplint's legal/copyright, build/include_alpha, and whitespace/newline categories have been suppressed.
  • Includes of the algorithm header are not required when using std::swap.
You can run checks easily by writing rules like the following in your makefile.
cpplint:
-./cpplint.py --linelength=100 \
$(shell find ./include/ -path "*.h") \
$(shell find ./sources/ -path "*.cpp" -or -path "*.h") 2>&1 | tee cpplint.log
In addition, nlib supports checking source code using clang-check. Generating a makefile with CMake creates a file called compile_commands.json, which clang-check uses to run.
For more information, see JSON Compilation Database Format Specification in the clang manual.

Automatic Formatting of Coding Style

You can also use clang-format to standardize the indentation and other stylistic aspects to suit your preferences. You can use the configuration used by nlib, which is stored in a .clang-format file (for clang 3.5) that is included in the nlib package. It is recommended that you first convert your source code en masse using clang-format, and then use cpplint.py to do common detailed checks.

Setting Warnings During Compilation

Code quality can be improved by testing with multiple compilers and runtime environments. Configure all compilers to relatively strict warning levels.
  • Ensure successful compilation with Visual Studio 2012 and all newer versions. In general, resolve all warnings shown when using the /W4 option.
    • Suppression of pragma warnings is permitted.
    • Make sure that the latest updates are always applied, and check _MSC_FULL_VER in your code.
    • Periodically reduce warnings using code analysis in Visual Studio.
  • Make sure that compilation succeeds using the latest versions of gcc and clang binary packages available for Cygwin (32-bit) and Ubuntu. In general, resolve all warnings shown when using -Wall -Wextra.
    • Always use the newest version of Cygwin. If a Cygwin update causes a build not to work as expected, revise the build scripts.
    • Suppression of pragma warnings is permitted.
    • Use -Wstack-usage=65536 to ensure that the stack is not overused.
    • Use -fstack-protector.
  • Make sure that compilation succeeds without warnings in the CAFE development environment.
  • Make sure that compilation succeeds in the CTR development environment.
  • It is preferable to have compilation succeed in all 64-bit environments.
  • Also resolve all Doxygen warnings.
The basic release conditions for nlib are ensuring that there are zero cpplint.py warnings and that compilation and tests succeed in all environments.

Additional Rules and Conventions

  • In general, code for functions that are not declared or implemented in functions defined in Platform.h must appear within code that can be compiled in all environments it could conceivably be used in. That said, inclusion of code that is optimized for specific hardware using SIMD instructions or other such techniques is permitted.
  • Use macros like NLIB_FINAL or NLIB_OVERRIDE and typedefs to safely incorporate C++11 features and continuously make it possible to port to C++11 or C++14.
  • Use POSIX.1 2008 error codes for error values whenever possible. Use the errno_t type, as recommended by https://www.securecoding.cert.org/confluence/x/QgAy.
    For example:
    typedef int errno_t;
    Using code like this makes it easier to understand that these are error values.
    • Values of POSIX.1 2008 error codes are sufficiently small integer values, so immediate values can be loaded efficiently on processors like ARM.
  • They must be implemented without using exceptions or RTTI. Make sure that your application code (that calls nlib) is written in a way that anticipates the use of these features, such as exception safety.
  • Instead of functions like malloc and free, use functions like nlib_malloc and nlib_free.
  • Use new (std::nothrow) for new.
  • The global new operator must not be overloaded within the library.
    • This overloading is not allowed because it affects the behavior of other libraries that you wouldn't think of as having dependencies.
  • Encode multi-byte strings in either ASCII or UTF-8.
    • The implementation must allow UTF-8, including in Windows. Path strings and strings passed to nlib_printf are passed to Win32 API after conversion to UTF-16 in nlib.
    • Command-line argument processing (Nflags) is an exception.
  • The char type must work correctly, whether signed or unsigned.
    • These can be handled explicitly as signed values by using signed char.
  • UTF-16 is supported if wchar_t is 16-bit; UTF-32 is supported if wchar_t is 32-bit.
    • NLIB_WCHAR_SIZE is defined as either 2 or 4, so you can use this to make the determination.
  • Use nlib_utf16_t as the type for UTF-16 characters, and use nlib_utf32_t as the type for UTF-32 characters.
    • In C++11, these types are defined with typedef as char16_t and char32_t, respectively.
  • In general, prioritize UTF-8 over UTF-16 and UTF-32.
    • UTF-8 consumes less memory, and is easy to support on multiple platforms.
    • The existence of surrogate pairs and combining characters make it impossible to always represent a single character with a fixed length, even if using UTF-16 or UTF-32.
    • Using nlib_strcplen_ex allows you to request the number of code points and byte length simultaneously while performing validation. If SIMD is supported, this operation can be sped up even more.
  • Move constructors and move assignment operators must be defined in code separated by NLIB_CXX11_RVALUE_REFERENCES, and you must also define member functions like the following so that the same functionality can be implemented in C++03.
    Type::Type(Type& rhs, ::nn::nlib::move_tag); // Behaves like the move constructor.
    Type& Type::assign(Type& rhs, ::nn::nlib::move_tag); // Behaves like the move assignment operator.
    Excellent alternative approaches like boost::move exist, but they cannot achieve full compatibility, so they are not used.
  • type_traits are used with care. This feature of C++11 is sometimes buggy or is not a complete implementation for some compilers or compiler versions. A tentative wrapper has been provided in nn/nlib/TypeTraits.h, but only use it if there is no other alternative.
  • Do not use non-copyable mixins.
    • Use NLIB_DISALLOW_COPY_AND_ASSIGN instead.
    • The increase in levels of inheritance can sometimes have a negative effect on source code analysis programs and documentation systems.
  • For the C standard library headers, include files in .h format. There are two reasons for this.
    • It handles situations when the C++ standard library headers are corrupted.
    • It minimizes code confusion caused by the inclusion of both %<xxx.h%> and %<cxxx%> formats within nlib. (The %<xxx.h%> format must be included because there is a requirement that nn/nlib/Platform.h can be compiled with a C compiler.)
  • Write tests in Google Test format, and make sure unit tests have been run on all platforms.
    • This makes it easier to find bugs that only occur in certain environments, in addition to platform-specific implementation bugs.
    • It is best if you separate the common tests to be run on any platform from the PC-specific tests.
      • This makes it easier to write stress tests and brute-force tests in environments with as much power as a PC.
      • On a PC environment, it is sometimes easy and convenient to use open-source software (OSS) in test programs.
      • In a Windows environment, DirectX and other such APIs can be used in test programs, which is sometimes convenient.
      • In environments where Google Test does not run, the testing library can be used.
      • When compiling using gcc, you can calculate test coverage automatically by using gcovr.
  • Write the Reference Manual in Doxygen format.
    • Doxygen is continuously being improved, and it is a documentation system that we can expect to exist and be improved even five years from now.
    • Doxygen comments are written within manual.h files located within each include directory. English versions of the manual are stored as manual_en.h. This approach allows multiple languages to be managed within the same repository.
  • If possible, enable building using CMake.
    • CMake is a representative selection for cases when you are coding in C/C++ that must run on both Windows and Linux.

Introduction to CMake

CMake can be used for building samples. CMake has been well-known for being adopted to build MySQL (MariaDB). Currently, it has established the position as the standard build system for various software supporting both Windows and Linux because it is written in C/C++. You may have noticed that many OSSs contain the file CMakeLists.txt. Using CMake allows you to generate makefiles for GNU make, Visual Studio and Xcode projects by writing this CMakefiles.txt script file.
The main advantages of using CMake are as follows.
  • It supports many systems and compilers.
    • It supports Windows, Linux, Mac OS X, FreeBSD, IRIX, AIX, Sun OS, HP-UX, and more.
    • It supports multiple IDEs, including all the Visual Studio versions, Eclipse CDT and Xcode.
    • You can also use Express Edition of Visual Studio and each of the latest versions of Visual Studio will be ready to be supported by the time when its RC version is released.
    • It can support cross-compilation with relatively simple settings.
  • Temporary files and other files created during builds don't clutter up the directories where the source files are located. This means Out-Of-Tree builds are possible. This function is always required when binary for multiple environments needs to be generated from one source tree.
  • Multiple independent projects can be generated from CMake and then used, which makes parallel builds easy.
  • It searches for the locations of required libraries and tools automatically.
  • The include path settings for libraries and links to libraries have been made cross-platform, so it is possible to write unified code even if the paths vary for each environment.
  • Installers can be created (CPack).
    • NSIS, deb, rpm, tgz, and so on.
  • Tests can be run and managed (CTest).
With nlib, using CMake allows you to generate solution files that work with all Visual Studio versions, makefiles for UNIX and Cygwin, and so on, and then build the program.
The ./cmake/toolchain-cafe.cmake and ./cmake/toolchain-ctr.cmake files within the package are configured for cross-compilation. This makes it possible to use CMake to generate makefiles even for CAFE/CTR sample programs, and then build the programs. The CAFE and CTR versions of nlib also have scripts for the standard make and omake build systems.

Cautions When Using CMake

  • A Windows version of CMake must be included to generate Visual Studio projects.
  • A Cygwin version of CMake must be installed to generate makefiles for Cygwin.
  • With CMake, Visual Studio and Xcode projects and makefiles must be output to separate directories for each Debug build and Release build.

FAQ

This section explains commonly encountered problems while using nlib, and their workarounds.
Overloading new/delete
nlib uses new (std::nothrow) for memory allocation. When applications and other libraries overload new or delete, operator new(size_t, const std::nothrow_t&) must also be overloaded.
For more information, see the heap_replace_malloc sample.
Overriding Functions Like malloc
nlib does not use malloc or free directly. Instead, it uses functions like nlib_malloc and nlib_free. These functions are weakly defined, so they can be overridden by user-defined functions.
You must override these functions if you want to use a user-defined heap. The following functions are required.
void* nlib_malloc(size_t size);
void nlib_free(void* ptr);
void* nlib_calloc(size_t nmemb, size_t size);
void* nlib_realloc(void* ptr, size_t size);
void* nlib_memalign(size_t alignment, size_t size); // Not defined if WIN32 or CTR.
For more information, see the heap_replace_malloc sample.
Compile Options
Platform-specific switches and other such definitions are defined in files such as cmake/common.cmake.win32 within the package. Please reference them.
Compilation results in "MACRO_XXXX not defined, compile may fail" being displayed.
You need to define MACRO_XXXX using the command-line arguments to the compiler.
Should we use the latest version for our project?
That's best left up to the judgment of the project using nlib. Writing acceptance tests for future use makes it easier to make this type of decision. However, only the latest version receives support. Note this fact.
Please release a special version of the package for our project.
There are as many project requirements as there are projects. For example, in the past, inquiries like the following have been received.
  • We don't need library X, so we'd like to have a special release with that removed.
  • We'd like a release without any samples.
  • We are using the 32-bit version of Visual Studio 2012, so we'd like a release that comes with a pre-built Visual Studio 2012 solution.
To handle such requirements in a flexible manner, the projects using nlib are permitted to restructure or repackage nlib and to redistribute it, but only within the project.
Why can't text encoded using Shift-JIS be displayed with nlib_printf?
The input string for nlib_printf must be encoded using UTF-8.
There is a bug in a version released half a year ago. Will you please release a patch that fixes it?
Bug fixes are applied to the latest version. You need to use a newer release that incorporates the fix.
Will you please make it follow our library or framework's approach?
If your approach is publicly available, and is supported by all platforms (Windows, Linux, FreeBSD, OS X, Cafe, and CTR), this might be possible. Please get in touch.

Revision History

2016-09-20

  • Started operation check in FreeBSD 11.0. FreeBSD 10.3 is no longer officially supported.
  • Updated the version of SQLite bundled with the OSS libraries to 3.1.4.1.
  • Implemented nlib_bitreverse32() and nlib_bitreverse64().
  • It is now possible to specify O_EXCL(NLIB_FD_O_EXCL) in nlib_fd_open()/tt>.
  • Modified nlib_fd_create() so that it fails if the file already exists.
  • nlib_strcplen2() was renamed nlib_strcplen(). nlib_strcplen2() will be provided as a macro for a while and then be deleted.
  • nlib_popcnt() was renamed nlib_popcnt32(). nlib_popcnt() will be provided as a macro for a while and then be deleted.
  • nlib_clz() was renamed nlib_clz32(). nlib_clz() will be provided as a macro for a while and then be deleted.
  • nlib_ctz() was renamed nlib_ctz32(). nlib_ctz() will be provided as a macro for a while and then be deleted.
  • Deleted deprecated files and functions.
  • Made other minor changes and fixes.

2016-08-23

  • Started operation check on Bash on Ubuntu on Windows. The compiled libraries are shipped with your PC version package.
  • Added heap::CentralHeap::FreeWithSize(), heap::CentralHeap::GetAllocSize(), and heap::CentralHeap::Realloc().
  • Added NMALLOC_QUERY_SET_COLOR, NMALLOC_QUERY_GET_COLOR, NMALLOC_QUERY_SET_NAME and NMALLOC_QUERY_GET_NAME to queries for nmalloc_query(). This function associates a 24-bit integer value or a string with any allocated region of 4096 bytes or more.
  • For how to use this function, see the object_tracking sample in the heap library.
  • Added NMALLOC_DUMP_PAGESUMMARY to NMallocDumpMode.
  • Improved improper behaviors of commands, including nmalloc, when failing to allocate a TLS index. nmalloc_setmark1(), nmalloc_setmark2(), nmalloc_getmark() and nmalloc_getobjptr() are now deprecated. They will be deleted in a future release.
  • NMALLOC_HEAPOPTION_CHECK_0 and NMALLOC_HEAPOPTION_CHECK_1 are now deprecated. They will be deleted in a future release.
  • Made other minor changes and fixes.

2016-07-26

2016-06-29

2016-05-31

  • Starting with the next release, a 64-bit cygwin build is provided as its PC version, and the library directory will be changed from lib/cmake/gcc to lib/cmake/cygwin.
  • Added nlib_remove() and nlib_thread_getname().
  • Improved the behavior of heap::CentralHeap(nmalloc) of the heap library.
    • Mitigated the issue where allocating a large size of memory (1 MB or more) caused fragmentation.
    • Mitigated the issue where an assigned memory address range was larger than necessary.
  • Other minor changes and fixes

2016-04-26

  • Started operation check in Alpine Linux 3.3.
  • Started operation check in Ubuntu 16.04LTS. Ubuntu 15.10 is no longer officially supported.
  • Started operation check in FreeBSD 10.3. FreeBSD 10.2 is no longer officially supported.
  • Mitigated an issue where allocating relatively large size memory (4 KB or more) using heap::CentralHeap(nmalloc) from the heap library caused fragmentation.
    • Allocation of a memory size up to 1 MB can now best fit within a free list (previously a size up to 512 KB can do so).
    • Fixed an issue where memory is prone to have fragmentations if the memory is allocated with an alignment.
  • Updated the version of SQLite to 3.12.2, which is included in the oss libraries. In addition, temporary files can be now automatically deleted.
  • In nlib_fd_pwrite() and nlib_fd_pread() for the CAFE and CTR versions, corrected the behavior found when an offset larger than the file size is specified.
  • simd::I64 is now deprecated.
  • The Crc32 class is now deprecated. Use nlib_crc32() or nlib_crc32c() instead.
  • Other minor changes and fixes

2016-03-29

2016-02-23

  • The oss library now uses nlib_malloc instead of nmalloc in platforms other than CAFE and CTR.
  • NMALLOC_QUERY_UNIFY_FREELIST can now be specified for the argument of nmalloc_query() in the heap library.
  • The fragmentation of metadata in the heap retained by the heap library's CentralHeap is reduced.
  • Added simd::I128::PopCntMask8(), simd::I128::ClzMask8(), and simd::I128::CtzMask8() to the simd library.
  • Corrected issues where some header files were missing.
  • Fixed an issue where nlib_malloc_size() could not be used in the Windows version of the DLL.
  • Made other minor changes and fixes.

2016-01-26

  • Started operation check in CentOS 7.2. Operation check in CentOS 7.1 is no longer supported.
  • Started operation check in Fedora 23. Operation check in Fedora 22 is no longer supported.
  • Added the nlib_is_error() function. It allows errors to be definitely evaluated and described for the return values of errno_t or bool.
  • The nlib_getenv() function now supports CAFE and CTR.
  • The IsOk() and IsError() functions for each class have been depreciated. Use the nlib_is_error() or operator bool() function instead.
  • Added the Init() member function to Utf16InputStream, Utf32InputStream, WcharInputStream, Base64InputStream and Base64OutputStream in the misc library.
  • The initialization process for the TextReader, TextWriter, BinaryReader and BinaryWriter classes for the misc library is now split into two stages: one with Init() and one with Open().
  • The IsOk(), IsError(), Initialize() and GetErrorValue() member functions for each class in the exi library have been depreciated.
  • The Initialize() member function for each class in the heap library has been depreciated.
  • The initialization process for the CsvReader class for the msgpack library is now split into two stages: one with Init() and one with Open().
  • Fixed an issue where nmalloc_aligned() crashed if memory was assigned with an alignment larger than 4096 bytes.
  • Corrected the issue where nmalloc_query(NMALLOC_QUERY_MAX_ALLOCATABLE_SIZE, ...) could return a size larger than the one actually allocated.
  • Corrected the issue where F128::StoreLoA4() and F128::StoreHiA4() of the simd library may crash during operation on the ARM processor.
  • Made other minor changes and fixes.
Contains Past Revision Histories.