Zhao Dongyang

OpenHarmony TPC

Not long ago, a new batch of JS/eTS components were added to the tripartite component library, including okio components. okio is an efficient IO library that can be applied to OpenAtom OpenHarmony (hereinafter referred to as “OpenHarmony”). It relies on system capabilities to provide encoding and decoding conversion capabilities for strings, read and write capabilities for basic data types, and support for reading and writing files. . This issue will introduce how okio works and how to use it.  

1. Background of okio
 

IO, that is, input and output (Input/Output). Most applications require data interaction with the outside world, which involves IO. The system provides IO capabilities. When using system IO, an intermediate buffer is usually required to save the read data. Data is first copied from the input stream buffer to the intermediate buffer, and then copied from the intermediate buffer to the output stream buffer. Multiple copies in the middle reduce IO efficiency and increase system consumption.In order to meet the higher requirements of developers for IO, the three-party component library launched an IO processing tool – okio (JS version). Okio uses Segment as a data storage container. By providing the ability to move, share, merge and split Segment, it makes data reading and writing very flexible, reduces data duplication, and improves IO efficiency. In addition, okio also recycles and reuses segments through SegmentPool, reducing the system consumption caused by creating a large number of segments.The following will take you to understand the working principle of the JS version of okio and explore how it can improve IO efficiency.  

2. Two basic concepts
 

Before deeply analyzing the working principle of okio, let’s first understand two basic concepts: Segment and SegmentPool.  1. SegmentOkio divides data into pieces and stores them in Segment.Segment is a real class for data storage, and internally maintains a byte array with a size of 8192 bytes for storing data. The minimum shareable and writable data size of a segment is 1024 bytes. Segment uses pos, limit, shared, owner, prev, and next to record read and write positions, whether it can be written, whether it can be shared, data owner, pre-node, and post-node information. Segment provides sharedCopy, unsharedCopy, split, push, pop, compact, writeTo and other interfaces to operate data.Segment has both front nodes and back nodes, forming a doubly linked list. When reading data, start reading from the head of the doubly linked list; and when writing data, write data from the end of the doubly linked list.  2. SegmentPoolIn order to manage Segment, okio maintains a Segment object pool (SegmentPool) to recycle, reuse and share memory for discarded Segment, thereby reducing memory application and GC (garbage collection, garbage collection) frequency, and optimizing performance.SegmentPool is a singly linked list consisting of up to 8 segments. The maximum size of a Segment is 8192 bytes (ie 8KB), so the maximum size of a SegmentPool is 64KB.  

3. The working principle of okio
 

The most important functions of okio components are “read” and “write”. Let’s start with reading and writing to understand the working principle of okio.  1. Read and write dataIn the process of reading and writing data, okio follows the principle of moving large blocks of data and copying small blocks of data.When okio reads data from the input stream to the input stream buffer, it will first find the Segment node at the end of the doubly linked list. If the remaining capacity of this node is sufficient, it will directly store the read data into this node. If the remaining capacity of this node is insufficient, take a Segment link from the SegmentPool to the end of the doubly linked list, and then store the data into this new node.Okio reads data from the input stream buffer and writes data to the output stream buffer. This process is more complicated, there are the following situations:(1) Get the Segment from the input stream buffer, if the data is full (the length of the byte array data is 8092 bytes), then directly modify the prev and next information of this Segment, and add it to the two-way output stream buffer The tail of the linked list saves a data copy process.  

Figure 1 Large block data movement  (2) Segment (assumed to be Segment1) is obtained from the input stream buffer. If the data is not full, the readable data of segment1 can be determined through pos and limit information, and then combined with the tail node of the doubly linked list of the output stream buffer ( Assume that it is Segment2) for comparison with the remaining capacity:

If the readable data of Segment1 is smaller than the remaining capacity of Segment2, copy the data of Segment1 to Segment2, and then recycle Segment1 to SegmentPool.

If the readable data of Segment1 is larger than the remaining capacity of Segment2, then directly modify the prev and next information of Segment1 and add it to the back of Segment2.

(3) Obtain the Segment (assumed to be Segment3) from the input stream buffer. If only part of the data needs to be transmitted (for example, the total data is 4096 bytes, only 1024 bytes are transmitted), okio will split the Segment3 into Segment3 through the split interface. Segment3-1 with 3072 bytes of data and Segment3-2 with 1024 bytes of data, and then write the data of Segment3-2 to the output stream buffer according to the logic of (2).

Figure 2 Segment split

When splitting a segment, you can specify the number of unread bytes (byteCount) contained in the first segment after splitting through parameters. After splitting, the data range contained in the first Segment is [pos, pos+byteCount), and the data range contained in the second Segment is [pos+byteCount, limit). When splitting a segment, the principle of moving large blocks of data and copying small blocks of data is also followed. When byteCount is greater than 1024, use the shared Segment, otherwise copy the data.

(Note: IO optimization related to files, streams, and sockets requires system support, which will be optimized and provided in subsequent versions.)

2. Segment recovery and reuse

Next, let’s take a look at how SegmentPool recycles and reuses Segments.

Every time okio wants to get a Segment, it gets it from the SegmentPool, and then puts it back into the SegmentPool for reuse after use. The core methods are take() and recycle().

(1) take () method

The take() method is responsible for obtaining available Segments from the head of the object pool singly linked list. If it cannot be obtained, it means that the singly linked list is empty. At this time, a new segment is created for the buffer. If it can be obtained, take out the head node of the singly-linked list, then set the next node as the head node of the singly-linked list, and empty the next of the taken out Segment, and update the size of the object pool at the same time.

(2) recycle() method

The recycle() method is responsible for recycling the used Segment in the buffer. When recycling starts, first update the size of the object pool, then add the recycled object Segment to the head of the singly linked list, and then reset the pos and limit of the Segment to 0. Note that Segments will not be recycled in the following situations:

The prev and next of the current segment are not empty

The current segment is shared

The object pool already has 8 segments

3. String processing

In addition to Segment and SegmentPool, okio also encapsulates the ByteString class for string processing. ByteString provides rich APIs such as Base64 encoding and decoding, utf-8 encoding and hexadecimal encoding and decoding, case conversion, content comparison, etc., which can easily process strings.

When processing strings, since ByteString holds the original string and the corresponding byte array at the same time, you can directly use the data in the byte array for operations without first converting the string to a byte array. Especially in scenarios where encoding is frequently converted, this way of exchanging space for time can avoid multiple conversions between character strings and byte arrays, reducing time and system performance consumption.

4. The use and examples of okio

1. Pre-configuration

Step 1: Add the following dependencies in the package.json file of entry.

"dependencies": {"okio": "^1.0.0"}

Step 2: Configure the mirror address of the warehouse.

npm config set @ohos:registry=https://repo.harmonyos.com/npm/

Step 3: Enter the following command in the Terminal of DevEco Studio to download the source code.

cd entrynpm install @ohos/okio

Step 4: Import the okio library into the header of the file.

import okio from '@ohos/okio';

Step 5: Apply for storage permission in the config.json file.

"reqPermissions": [{"name": "ohos.permission.WRITE_USER_STORAGE", //Permission to write user storage"reason": "Storage","usedScene": {"when": "always","ability": ["com.example.okioapplicaTIon.MainAbility"}},{"name": "ohos.permission.READ_USER_STORAGE", //Read user storage permission"reason": "Storage","usedScene": {"when": "always","ability": ["com.example.okioapplicaTIon.MainAbility"]}},{"name": "ohos.permission.WRITE_EXTERNAL_MEDIA_MEMORY", //Permission to write to external storage"reason": "Storage","usedScene": {"when": "always","ability": ["com.example.okioapplicaTIon.MainAbility"]}}]

 

2. Code implementation

After performing the above configuration operations, you can enter the code writing stage. Developers can use the rich API interfaces provided by okio to realize functions. Here are four implementation examples for your reference and study.

Example 1: File writing and reading

This example writes content to a file through sink and reads content from a file through source. code show as below:

//Write the content to the file through the sinkvar sink = new okio.Sink(this.fileUri);sink.write(this.Value,false);//Read content from file through sourcevar source = new okio.Source(this.fileUri);source.read().then(function (data) {context.readValue = data;}).catch(function (error){console.log("error=>"+error);});

 

Example 2: Base64 decoding

This example implements the Base64 decoding function through ByteString, the code is as follows:

let byteStringObj = new okio.ByteString.ByteString(''); //Generate ByteString objectlet decodeBase64 = byteStringObj.decodeBase64('SGVsbG8gd29ybGQ='); //decode Base64 stringthis.decodeBase64Value = JSON.stringify(decodeBase64); //Display the decoding result

 

Example 3: Hexadecimal decoding

This example implements the hexadecimal decoding function through ByteString, the code is as follows:

let byteStringObj = new okio.ByteString.ByteString('');let decodehex = byteStringObj.decodeHex('48656C6C6F20776F726C640D0A');this.decodeHexValue = JSON.stringify(decodehex);

 

Example 4: Utf8 encoding

This example implements the Utf8 encoding function through ByteString, the code is as follows:

let byteStringObj = new okio.ByteString.ByteString('');let encodeUtf8 = byteStringObj.encodeUtf8('Hello world #4  ( ͡ㆆ ͜ʖ ͡ㆆ)');this.encodeUtf8Value = JSON.stringify(encodeUtf8);

 

This issue of okio components is introduced here. The okio component has been open sourced, and everyone is welcome to contribute. The open source address is as follows:

https://gitee.com/openharmony-tpc/okio

Leave a Reply

Your email address will not be published.