Hadoop HDFS3.0的默认配置项

description

version of this

configuration file

RPC

address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist,

    the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1

    dfs.namenode.rpc-address.EXAMPLENAMESERVICE

    The value of this property will take the form of nn-host1:rpc-port. The NameNode's default RPC port is 9820.

The

actual address the RPC server will bind to. If this optional address is

    set, it overrides only the hostname portion of dfs.namenode.rpc-address.

    It can also be specified per name node or name service for HA/Federation.

    This is useful for making the name node listen on all interfaces by

    setting it to 0.0.0.0.

RPC

address for HDFS Services communication. BackupNode, Datanodes and all other services should be

    connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist,

    the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns1

    dfs.namenode.rpc-address.EXAMPLENAMESERVICE

    The value of this property will take the form of nn-host1:rpc-port.

    If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default.

The

actual address the service RPC server will bind to. If this optional address is

    set, it overrides only the hostname portion of dfs.namenode.servicerpc-address.

    It can also be specified per name node or name service for HA/Federation.

    This is useful for making the name node listen on all interfaces by

    setting it to 0.0.0.0.

NameNode

RPC lifeline address.  This is an optional separate RPC address

    that can be used to isolate health checks and liveness to protect against

    resource exhaustion in the main RPC handler pool.  In the case of

    HA/Federation where multiple NameNodes exist, the name service ID is added

    to the name e.g. dfs.namenode.lifeline.rpc-address.ns1.  The value of this

    property will take the form of nn-host1:rpc-port.  If this property is not

    defined, then the NameNode will not start a lifeline RPC server.  By

    default, the property is not defined.

The

actual address the lifeline RPC server will bind to.  If this optional

    address is set, it overrides only the hostname portion of

    dfs.namenode.lifeline.rpc-address.  It can also be specified per name node

    or name service for HA/Federation.  This is useful for making the name node

    listen on all interfaces by setting it to 0.0.0.0.

The secondary

namenode http server address and port.

The secondary

namenode HTTPS server address and port.

The datanode server

address and port for data transfer.

The datanode http

server address and port.

The datanode ipc

server address and port.

The number of server

threads for the datanode.

The address and the

base port where the dfs namenode web ui will listen on.

The

actual address the HTTP server will bind to. If this optional address

    is set, it overrides only the hostname portion of dfs.namenode.http-address.

    It can also be specified per name node or name service for HA/Federation.

    This is useful for making the name node HTTP server listen on all

    interfaces by setting it to 0.0.0.0.

This

time decides the interval to check for expired datanodes.

    With this value and dfs.heartbeat.interval, the interval of

    deciding the datanode is stale or not is also calculated.

    The unit of this configuration is millisecond.

Decide

if HTTPS(SSL) is supported on HDFS

    This configures the HTTP endpoint for HDFS daemons:

      The following values are supported:

      - HTTP_ONLY : Service is provided only on http

      - HTTPS_ONLY : Service is provided only on https

      - HTTP_AND_HTTPS : Service is provided both on http and https

Whether SSL client

certificate authentication is required

The

number of times the HDFS client will pull a socket from the

   cache.  Once this number is exceeded, the client will try to create a new

   socket.

Resource

file from which ssl server keystore

  information will be extracted

Resource

file from which ssl client keystore

  information will be extracted

The datanode secure

http server address and port.

The namenode secure

http server address and port.

The

actual address the HTTPS server will bind to. If this optional address

    is set, it overrides only the hostname portion of dfs.namenode.https-address.

    It can also be specified per name node or name service for HA/Federation.

    This is useful for making the name node HTTPS server listen on all

    interfaces by setting it to 0.0.0.0.

The

name of the Network Interface from which a data node should

     report its IP address. e.g. eth2. This setting may be required for some

     multi-homed nodes where the DataNodes are assigned multiple hostnames

     and it is desirable for the DataNodes to use a non-default hostname.

     Prefer using hadoop.security.dns.interface over

     dfs.datanode.dns.interface.

The

host name or IP address of the name server (DNS) which a DataNode

    should use to determine its own host name.

    Prefer using hadoop.security.dns.nameserver over

    dfs.datanode.dns.nameserver.

The

backup node server address and port.

    If the port is 0 then the server will start on a free port.

The

backup node http server address and port.

    If the port is 0 then the server will start on a free port.

Decide if

chooseTarget considers the target's load or not

The

factor by which a node's load can exceed the average

      before being rejected for writes, only if considerLoad is true.

The number of bytes

to view for a file on the browser.

Reserved

space in bytes per volume. Always leave this much space free for non dfs use.

      Specific storage type based reservation is also supported. The property can be followed with

      corresponding storage types ([ssd]/[disk]/[archive]/[ram_disk]) for cluster with heterogeneous storage.

      For example, reserved space for RAM_DISK storage can be configured using property

      'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured

      then dfs.datanode.du.reserved will be used.

Determines

where on the local filesystem the DFS name node

      should store the name table(fsimage).  If this is a comma-delimited list

      of directories then the name table is replicated in all of the

      directories, for redundancy.

Set

to true to enable NameNode to attempt recovering a

      previously failed dfs.namenode.name.dir. When enabled, a recovery of any

      failed directory is attempted during checkpoint.

Defines

the maximum number of bytes in UTF-8 encoding in each

      component of a path.  A value of 0 will disable the check.

Defines

the maximum number of items that a directory may

      contain. Cannot set the property to a value less than 1 or more than

      6400000.

Minimum

block size in bytes, enforced by the Namenode at create

      time. This prevents the accidental creation of files with tiny block

      sizes (and thus many blocks), which can degrade

      performance.

Maximum

number of blocks per file, enforced by the Namenode on

        write. This prevents the creation of extremely large files which can

        degrade performance.

Determines

where on the local filesystem the DFS name node

      should store the transaction (edits) file. If this is a comma-delimited list

      of directories then the transaction file is replicated in all of the

      directories, for redundancy. Default value is same as dfs.namenode.name.dir

This

should be a subset of dfs.namenode.edits.dir,

      to ensure that the transaction (edits) file

      in these places is always up-to-date.

A

directory on shared storage between the multiple namenodes

  in an HA cluster. This directory will be written by the active and read

  by the standby in order to keep the namespaces synchronized. This directory

  does not need to be listed in dfs.namenode.edits.dir above. It should be

  left empty in a non-HA cluster.

If

"true", enable permission checking in HDFS.

    If "false", permission checking is turned off,

    but all other behavior is unchanged.

    Switching from one parameter value to the other does not change the mode,

    owner or group of files or directories.

The

name of the group of super-users.

    The value should be a single group name.

ACL

for the admins, this configuration is used to control

     who can access the default servlets in the namenode, etc. The value

     should be a comma separated list of users and groups. The user list

     comes first and is separated by a space followed by the group list,

     e.g. "user1,user2 group1,group2". Both users and groups are optional,

     so "user1", " group1", "", "user1 group1", "user1,user2 group1,group2"

     are all valid (note the leading space in " group1"). '*' grants access

     to all users and groups, e.g. '*', '* ' and ' *' are all valid.

Set

to true to enable support for HDFS ACLs (Access Control Lists).  By

    default, ACLs are disabled.  When ACLs are disabled, the NameNode rejects

    all RPCs related to setting or getting ACLs.

Set

to true to enable POSIX style ACL inheritance. When it is enabled

      and the create request comes from a compatible client, the NameNode

      will apply default ACLs from the parent directory to the create mode

      and ignore the client umask. If no default ACL found, it will apply the

      client umask.

The

NameNode periodically scans the namespace for LazyPersist files with

    missing blocks and unlinks them from the namespace. This configuration key

    controls the interval between successive scans. Set it to a negative value

    to disable this behavior.

If

"true", access tokens are used as capabilities for accessing datanodes.

    If "false", no access tokens are checked on accessing datanodes.

Interval in minutes

at which namenode updates its access keys.

The lifetime of

access tokens in minutes.

If

"true", block tokens are written using Protocol Buffers.

    If "false", block tokens are written using Legacy format.

Determines

where on the local filesystem an DFS data node

  should store its blocks.  If this is a comma-delimited

  list of directories, then data will be stored in all named

  directories, typically on different devices. The directories should be tagged

  with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS

  storage policies. The default storage type will be DISK if the directory does

  not have a storage type tagged explicitly. Directories that do not exist will

  be created if local filesystem permission allows.

Permissions

for the directories on on the local filesystem where

  the DFS data node store its blocks. The permissions can either be octal or

  symbolic.

Default

block replication.

  The actual number of replications can be specified when the file is created.

  The default is used if replication is not specified in create time.

Maximal block

replication.

Minimal block

replication.

Minimal live block

replication in existence of maintenance mode.

a

separate minimum replication factor for calculating safe block count.

      This is an expert level setting.

      Setting this lower than the dfs.namenode.replication.min

      is not recommend and/or dangerous for production setups.

      When it's not set it takes value from dfs.namenode.replication.min

The

default block size for new files, in bytes.

      You can use the following suffix (case insensitive):

      k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.),

      Or provide complete size in bytes (such as 134217728 for 128 MB).

The

number of retries for writing blocks to the data nodes,

  before we signal failure to the application.

If

there is a datanode/network failure in the write pipeline,

    DFSClient will try to remove the failed datanode from the pipeline

    and then continue writing with the remaining datanodes. As a result,

    the number of datanodes in the pipeline is decreased.  The feature is

    to add new datanodes to the pipeline.

    This is a site-wide property to enable/disable the feature.

    When the cluster size is extremely small, e.g. 3 nodes or less, cluster

    administrators may want to set the policy to NEVER in the default

    configuration file or disable this feature.  Otherwise, users may

    experience an unusually high rate of pipeline failures since it is

    impossible to find new datanodes for replacement.

    See also dfs.client.block.write.replace-datanode-on-failure.policy

This

property is used only if the value of

    dfs.client.block.write.replace-datanode-on-failure.enable is true.

    ALWAYS: always add a new datanode when an existing datanode is removed.

    NEVER: never add a new datanode.

    DEFAULT:

      Let r be the replication number.

      Let n be the number of existing datanodes.

      Add a new datanode only if r is greater than or equal to 3 and either

      (1) floor(r/2) is greater than or equal to n; or

      (2) r is greater than n and the block is hflushed/appended.

This

property is used only if the value of

    dfs.client.block.write.replace-datanode-on-failure.enable is true.

    Best effort means that the client will try to replace a failed datanode

    in write pipeline (provided that the policy is satisfied), however, it

    continues the write operation in case that the datanode replacement also

    fails.

    Suppose the datanode replacement fails.

    false: An exception should be thrown so that the write will fail.

    true : The write should be resumed with the remaining datandoes.

    Note that setting this property to true allows writing to a pipeline

    with a smaller number of datanodes.  As a result, it increases the

    probability of data loss.

The

minimum number of replications that are needed to not to fail

      the write pipeline if new datanodes can not be found to replace

      failed datanodes (could be due to network failure) in the write pipeline.

      If the number of the remaining datanodes in the write pipeline is greater

      than or equal to this property value, continue writing to the remaining nodes.

      Otherwise throw exception.

      If this is set to 0, an exception will be thrown, when a replacement

      can not be found.

      See also dfs.client.block.write.replace-datanode-on-failure.policy

Determines block

reporting interval in milliseconds.

Delay

for first block report in seconds. Support multiple time unit

    suffix(case insensitive), as described in dfs.heartbeat.interval.

If

the number of blocks on the DataNode is below this

    threshold then it will send block reports for all Storage Directories

    in a single message.

    If the number of blocks exceeds this threshold then the DataNode will

    send block reports for each Storage Directory in separate messages.

    Set to zero to always split.

The

maximum number of leases for full block reports that the

    NameNode will issue at any given time.  This prevents the NameNode from

    being flooded with full block reports that use up all the RPC handler

    threads.  This number should never be more than the number of RPC handler

    threads or less than 1.

The

number of milliseconds that the NameNode will wait before invalidating

    a full block report lease.  This prevents a crashed DataNode from

    permanently using up a full block report lease.

Interval

in seconds for Datanode to scan data directories and

  reconcile the difference between blocks in memory and on the disk.

  Support multiple time unit suffix(case insensitive), as described

  in dfs.heartbeat.interval.

How

many threads should the threadpool used to compile reports

  for volumes in parallel have.

The

report compilation threads are limited to only running for

  a given number of milliseconds per second, as configured by the

  property. The limit is taken per thread, not in aggregate, e.g. setting

  a limit of 100ms for 4 compiler threads will result in each thread being

  limited to 100ms, not 25ms.

  Note that the throttle does not interrupt the report compiler threads, so the

  actual running time of the threads per second will typically be somewhat

  higher than the throttle limit, usually by no more than 20%.

  Setting this limit to 1000 disables compiler thread throttling. Only

  values between 1 and 1000 are valid. Setting an invalid value will result

  in the throttle being disabled and an error message being logged. 1000 is

  the default setting.

Determines

datanode heartbeat interval in seconds.

    Can use the following suffix (case insensitive):

    ms(millis), s(sec), m(min), h(hour), d(day)

    to specify the time (such as 2s, 2m, 1h, etc.).

    Or provide complete number in seconds (such as 30 for 30 seconds).

Sets

the interval in seconds between sending DataNode Lifeline Protocol

    messages from the DataNode to the NameNode.  The value must be greater than

    the value of dfs.heartbeat.interval.  If this property is not defined, then

    the default behavior is to calculate the interval as 3x the value of

    dfs.heartbeat.interval.  Note that normal heartbeat processing may cause the

    DataNode to postpone sending lifeline messages if they are not required.

    Under normal operations with speedy heartbeat processing, it is possible

    that no lifeline messages will need to be sent at all.  This property has no

    effect if dfs.namenode.lifeline.rpc-address is not defined.

The

number of Namenode RPC server threads that listen to

  requests from clients.

  If dfs.namenode.servicerpc-address is not configured then

  Namenode RPC server threads listen to requests from all nodes.

The

number of Namenode RPC server threads that listen to

  requests from DataNodes and from all other non-client nodes.

  dfs.namenode.service.handler.count will be valid only if

  dfs.namenode.servicerpc-address is configured.

A

ratio applied to the value of dfs.namenode.handler.count, which then

    provides the number of RPC server threads the NameNode runs for handling the

    lifeline RPC server.  For example, if dfs.namenode.handler.count is 100, and

    dfs.namenode.lifeline.handler.factor is 0.10, then the NameNode starts

    100 * 0.10 = 10 threads for handling the lifeline RPC server.  It is common

    to tune the value of dfs.namenode.handler.count as a function of the number

    of DataNodes in a cluster.  Using this property allows for the lifeline RPC

    server handler threads to be tuned automatically without needing to touch a

    separate property.  Lifeline message processing is lightweight, so it is

    expected to require many fewer threads than the main NameNode RPC server.

    This property is not used if dfs.namenode.lifeline.handler.count is defined,

    which sets an absolute thread count.  This property has no effect if

    dfs.namenode.lifeline.rpc-address is not defined.

Sets

an absolute number of RPC server threads the NameNode runs for handling

    the DataNode Lifeline Protocol and HA health check requests from ZKFC.  If

    this property is defined, then it overrides the behavior of

    dfs.namenode.lifeline.handler.ratio.  By default, it is not defined.  This

    property has no effect if dfs.namenode.lifeline.rpc-address is not defined.

Specifies

the percentage of blocks that should satisfy

    the minimal replication requirement defined by dfs.namenode.replication.min.

    Values less than or equal to 0 mean not to wait for any particular

    percentage of blocks before exiting safemode.

    Values greater than 1 will make safe mode permanent.

Specifies

the number of datanodes that must be considered alive

    before the name node exits safemode.

    Values less than or equal to 0 mean not to take the number of live

    datanodes into account when deciding whether to remain in safe mode

    during startup.

    Values greater than the number of datanodes in the cluster

    will make safe mode permanent.

Determines

extension of safe mode in milliseconds after the threshold level

    is reached.  Support multiple time unit suffix (case insensitive), as

    described in dfs.heartbeat.interval.

The

interval in milliseconds at which the NameNode resource checker runs.

    The checker calculates the number of the NameNode storage volumes whose

    available spaces are more than dfs.namenode.resource.du.reserved, and

    enters safemode if the number becomes lower than the minimum value

    specified by dfs.namenode.resource.checked.volumes.minimum.

The

amount of space to reserve/require for a NameNode storage directory

    in bytes. The default is 100MB.

A

list of local directories for the NameNode resource checker to check in

    addition to the local edits directories.

The minimum number of

redundant NameNode storage volumes required.

Specifies

the maximum amount of bandwidth that each datanode

        can utilize for the balancing purpose in term of

        the number of bytes per second. You can use the following

        suffix (case insensitive):

        k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa)to specify the size

        (such as 128k, 512m, 1g, etc.).

        Or provide complete size in bytes (such as 134217728 for 128 MB).

Names

a file that contains a list of hosts that are

  permitted to connect to the namenode. The full pathname of the file

  must be specified.  If the value is empty, all hosts are

  permitted.

Names

a file that contains a list of hosts that are

  not permitted to connect to the namenode.  The full pathname of the

  file must be specified.  If the value is empty, no hosts are

  excluded.

The

maximum number of files, directories and blocks

  dfs supports. A value of zero indicates no limit to the number

  of objects that dfs supports.

If

true (the default), then the namenode requires that a connecting

    datanode's address must be resolved to a hostname.  If necessary, a reverse

    DNS lookup is performed.  All attempts to register a datanode from an

    unresolvable address are rejected.

    It is recommended that this setting be left on to prevent accidental

    registration of datanodes listed by hostname in the excludes file during a

    DNS outage.  Only set this to false in environments where there is no

    infrastructure to support reverse DNS lookup.

Namenode

periodicity in seconds to check if

    decommission or maintenance is complete. Support multiple time unit

    suffix(case insensitive), as described in dfs.heartbeat.interval.

The

approximate number of blocks to process per decommission

    or maintenance interval, as defined in dfs.namenode.decommission.interval.

The

maximum number of decommission-in-progress or

    entering-maintenance datanodes nodes that will be tracked at one time by

    the namenode. Tracking these datanode consumes additional NN memory

    proportional to the number of blocks on the datnode. Having a conservative

    limit reduces the potential impact of decommissioning or maintenance of

    a large number of nodes at once.

    A value of 0 means no limit will be enforced.

The

periodicity in seconds with which the namenode computes

  low redundancy work for datanodes. Support multiple time unit suffix(case insensitive),

  as described in dfs.heartbeat.interval.

The

access time for HDFS file is precise upto this value.

               The default value is 1 hour. Setting a value of 0 disables

               access times for HDFS.

Comma-separated list

of datanode plug-ins to be activated.

Comma-separated list

of namenode plug-ins to be activated.

Controls

how the default block placement policy places

  the first replica of a block. When true, it will prefer the node where

  the client is running.  When false, it will prefer a node in the same rack

  as the client. Setting to false avoids situations where entire copies of

  large files end up on a single node, thus creating hotspots.

The

size of buffer to stream files.

  The size of this buffer should probably be a multiple of hardware

  page size (4096 on Intel x86), and it determines how much data is

  buffered during read and write operations.

The

number of bytes per checksum.  Must not be larger than

  dfs.stream-buffer-size

Packet size for

clients to write

The

maximum period to keep a DN in the excluded nodes list

  at a client. After this period, in milliseconds, the previously excluded node(s) will

  be removed automatically from the cache and will be considered good for block allocations

  again. Useful to lower or raise in situations where you keep a file open for very long

  periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance

  restarts. Defaults to 10 minutes.

Determines

where on the local filesystem the DFS secondary

      name node should store the temporary images to merge.

      If this is a comma-delimited list of directories then the image is

      replicated in all of the directories for redundancy.

Determines

where on the local filesystem the DFS secondary

      name node should store the temporary edits to merge.

      If this is a comma-delimited list of directories then the edits is

      replicated in all of the directories for redundancy.

      Default value is same as dfs.namenode.checkpoint.dir

The

number of seconds between two periodic checkpoints.

    Support multiple time unit suffix(case insensitive), as described

    in dfs.heartbeat.interval.

The

Secondary NameNode or CheckpointNode will create a checkpoint

  of the namespace every 'dfs.namenode.checkpoint.txns' transactions, regardless

  of whether 'dfs.namenode.checkpoint.period' has expired.

The

SecondaryNameNode and CheckpointNode will poll the NameNode

  every 'dfs.namenode.checkpoint.check.period' seconds to query the number

  of uncheckpointed transactions. Support multiple time unit suffix(case insensitive),

  as described in dfs.heartbeat.interval.

The

SecondaryNameNode retries failed checkpointing. If the

  failure occurs while loading fsimage or replaying edits, the number of

  retries is limited by this variable.

Used

to calculate the amount of time between retries when in the 'quiet' period

    for creating checkpoints (active namenode already has an up-to-date image from another

    checkpointer), so we wait a multiplier of the dfs.namenode.checkpoint.check.period before

    retrying the checkpoint because another node likely is already managing the checkpoints,

    allowing us to save bandwidth to transfer checkpoints that don't need to be used.

The

number of image checkpoint files (fsimage_*) that will be retained by

  the NameNode and Secondary NameNode in their storage directories. All edit

  logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained

  checkpoint will also be retained.

The

number of extra transactions which should be retained

  beyond what is minimally necessary for a NN restart.

  It does not translate directly to file's age, or the number of files kept,

  but to the number of transactions (here "edits" means transactions).

  One edit file may contain several transactions (edits).

  During checkpoint, NameNode will identify the total number of edits to retain as extra by

  checking the latest checkpoint transaction value, subtracted by the value of this property.

  Then, it scans edits files to identify the older ones that don't include the computed range of

  retained transactions that are to be kept around, and purges them subsequently.

  The retainment can be useful for audit purposes or for an HA setup where a remote Standby Node may have

  been offline for some time and need to have a longer backlog of retained

  edits in order to start again.

  Typically each edit is on the order of a few hundred bytes, so the default

  of 1 million edits should be on the order of hundreds of MBs or low GBs.

  NOTE: Fewer extra edits may be retained than value specified for this setting

  if doing so would mean that more segments would be retained than the number

  configured by dfs.namenode.max.extra.edits.segments.retained.

The

maximum number of extra edit log segments which should be retained

  beyond what is minimally necessary for a NN restart. When used in conjunction with

  dfs.namenode.num.extra.edits.retained, this configuration property serves to cap

  the number of extra edits files to a reasonable value.

The

update interval for master key for delegation tokens

       in the namenode in milliseconds.

The

maximum lifetime in milliseconds for which a delegation

      token is valid.

The renewal interval

for delegation token in milliseconds.

The

number of volumes that are allowed to

  fail before a datanode stops offering service. By default

  any volume failure will cause a datanode to shutdown.

Should the dfs image

be compressed?

If

the dfs image is compressed, how should they be compressed?

               This has to be a codec defined in io.compression.codecs.

Socket

timeout for image transfer in milliseconds. This timeout and the related

        dfs.image.transfer.bandwidthPerSec parameter should be configured such

        that normal image transfer can complete successfully.

        This timeout prevents client hangs when the sender fails during

        image transfer. This is socket timeout during image transfer.

Maximum

bandwidth used for regular image transfers (instead of

        bootstrapping the standby namenode), in bytes per second.

        This can help keep normal namenode operations responsive during

        checkpointing. The maximum bandwidth and timeout in

        dfs.image.transfer.timeout should be set such that normal image

        transfers can complete successfully.

        A default value of 0 indicates that throttling is disabled.

        The maximum bandwidth used for bootstrapping standby namenode is

        configured with dfs.image.transfer-bootstrap-standby.bandwidthPerSec.

Maximum

bandwidth used for transferring image to bootstrap standby

      namenode, in bytes per second.

      A default value of 0 indicates that throttling is disabled. This default

      value should be used in most cases, to ensure timely HA operations.

      The maximum bandwidth used for regular image transfers is configured

      with dfs.image.transfer.bandwidthPerSec.

Chunksize

in bytes to upload the checkpoint.

        Chunked streaming is used to avoid internal buffering of contents

        of image file of huge size.

Socket

timeout for edit log transfer in milliseconds. This timeout

    should be configured such that normal edit log transfer for journal

    node syncing can complete successfully.

Maximum

bandwidth used for transferring edit log to between journal nodes

    for syncing, in bytes per second.

    A default value of 0 indicates that throttling is disabled.

Does

HDFS namenode allow itself to be formatted?

               You may consider setting this to false for any production

               cluster, to avoid any possibility of formatting a running DFS.

Specifies

the maximum number of threads to use for transferring data

        in and out of the DN.

If

this is positive, the DataNode will not scan any

        individual block more than once in the specified scan period.

        If this is negative, the block scanner is disabled.

        If this is set to zero, then the default value of 504 hours

        or 3 weeks is used. Prior versions of HDFS incorrectly documented

        that setting this key to zero will disable the block scanner.

If

this is 0, the DataNode's block scanner will be disabled.  If this

        is positive, this is the number of bytes per second that the DataNode's

        block scanner will try to scan from each volume.

While

reading block files, if the Hadoop native libraries are available,

        the datanode can use the posix_fadvise system call to explicitly

        page data into the operating system buffer cache ahead of the current

        reader's position. This can improve performance especially when

        disks are highly contended.

        This configuration specifies the number of bytes ahead of the current

        read position which the datanode will attempt to read ahead. This

        feature may be disabled by configuring this property to 0.

        If the native libraries are not available, this configuration has no

        effect.

In

some workloads, the data read from HDFS is known to be significantly

        large enough that it is unlikely to be useful to cache it in the

        operating system buffer cache. In this case, the DataNode may be

        configured to automatically purge all data from the buffer cache

        after it is delivered to the client. This behavior is automatically

        disabled for workloads which read only short sections of a block

        (e.g HBase random-IO workloads).

        This may improve performance for some workloads by freeing buffer

        cache space usage for more cacheable data.

        If the Hadoop native libraries are not available, this configuration

        has no effect.

In

some workloads, the data written to HDFS is known to be significantly

        large enough that it is unlikely to be useful to cache it in the

        operating system buffer cache. In this case, the DataNode may be

        configured to automatically purge all data from the buffer cache

        after it is written to disk.

        This may improve performance for some workloads by freeing buffer

        cache space usage for more cacheable data.

        If the Hadoop native libraries are not available, this configuration

        has no effect.

If

this configuration is enabled, the datanode will instruct the

        operating system to enqueue all written data to the disk immediately

        after it is written. This differs from the usual OS policy which

        may wait for up to 30 seconds before triggering writeback.

        This may improve performance for some workloads by smoothing the

        IO profile for data written to disk.

        If the Hadoop native libraries are not available, this configuration

        has no effect.

Expert

only. The number of client failover attempts that should be

    made before the failover is considered failed.

Expert

only. The time to wait, in milliseconds, between failover

    attempts increases exponentially as a function of the number of

    attempts made so far, with a random factor of +/- 50%. This option

    specifies the base value used in the failover calculation. The

    first failover will retry immediately. The 2nd failover attempt

    will delay at least dfs.client.failover.sleep.base.millis

    milliseconds. And so on.

Expert

only. The time to wait, in milliseconds, between failover

    attempts increases exponentially as a function of the number of

    attempts made so far, with a random factor of +/- 50%. This option

    specifies the maximum value to wait between failovers.

    Specifically, the time between two failover attempts will not

    exceed +/- 50% of dfs.client.failover.sleep.max.millis

    milliseconds.

Expert

only. Indicates the number of retries a failover IPC client

    will make to establish a server connection.

Expert

only. The number of retry attempts a failover IPC client

    will make on socket timeout when establishing a server connection.

Expert

only. The time to wait, in seconds, from reception of an

    datanode shutdown notification for quick restart, until declaring

    the datanode dead and invoking the normal recovery mechanisms.

    The notification is sent by a datanode when it is being shutdown

    using the shutdownDatanode admin command with the upgrade option.

    Support multiple time unit suffix(case insensitive), as described

    in dfs.heartbeat.interval.

Comma-separated list

of nameservices.

The

ID of this nameservice. If the nameservice ID is not

    configured or more than one nameservice is configured for

    dfs.nameservices it is determined automatically by

    matching the local node's address with the configured address.

Comma-separated

list of nameservices that belong to this cluster.

    Datanode will report to all the nameservices in this list. By default

    this is set to the value of dfs.nameservices.

The

prefix for a given nameservice, contains a comma-separated

    list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).

    Unique identifiers for each NameNode in the nameservice, delimited by

    commas. This will be used by DataNodes to determine all the NameNodes

    in the cluster. For example, if you used “mycluster” as the nameservice

    ID previously, and you wanted to use “nn1” and “nn2” as the individual

    IDs of the NameNodes, you would configure a property

    dfs.ha.namenodes.mycluster, and its value "nn1,nn2".

The

ID of this namenode. If the namenode ID is not configured it

    is determined automatically by matching the local node's address

    with the configured address.

How

often, in seconds, the StandbyNode should ask the active to

    roll edit logs. Since the StandbyNode only reads from finalized

    log segments, the StandbyNode will only be as up-to-date as how

    often the logs are rolled. Note that failover triggers a log roll

    so the StandbyNode will be up to date before it becomes active.

    Support multiple time unit suffix(case insensitive), as described

    in dfs.heartbeat.interval.

How

often, in seconds, the StandbyNode should check for new

    finalized log segments in the shared edits log.

    Support multiple time unit suffix(case insensitive), as described

    in dfs.heartbeat.interval.

Number of retries to

use when contacting the namenode when tailing the log.

The timeout in

seconds of calling rollEdits RPC on Active NN.

Whether

automatic failover is enabled. See the HDFS High

    Availability documentation for details on automatic HA

    configuration.

Whether

clients should use datanode hostnames when

    connecting to datanodes.

Whether

datanodes should use datanode hostnames when

    connecting to other datanodes for data transfer.

A

comma separated list of network interface names to use

    for data transfer between the client and datanodes. When creating

    a connection to read from or write to a datanode, the client

    chooses one of the specified interfaces at random and binds its

    socket to the IP of that interface. Individual names may be

    specified as either an interface name (eg "eth0"), a subinterface

    name (eg "eth0:0"), or an IP address (which may be specified using

    CIDR notation to match a range of IPs).

A

comma-separated list of paths to use when creating file descriptors that

    will be shared between the DataNode and the DFSClient.  Typically we use

    /dev/shm, so that the file descriptors will not be written to disk.

    Systems that don't have /dev/shm will fall back to /tmp by default.

The

length of time in milliseconds that the short-circuit shared memory

    watcher will go between checking for java interruptions sent from other

    threads.  This is provided mainly for unit tests.

The

NameNode service principal. This is typically set to

    nn/[email protected]. Each NameNode will substitute _HOST with its

    own fully qualified hostname at startup. The _HOST placeholder

    allows using the same configuration setting on both NameNodes

    in an HA setup.

The

keytab file used by each NameNode daemon to login as its

    service principal. The principal name is configured with

    dfs.namenode.kerberos.principal.

The

DataNode service principal. This is typically set to

    dn/[email protected]. Each DataNode will substitute _HOST with its

    own fully qualified hostname at startup. The _HOST placeholder

    allows using the same configuration setting on all DataNodes.

The

keytab file used by each DataNode daemon to login as its

    service principal. The principal name is configured with

    dfs.datanode.kerberos.principal.

The

JournalNode service principal. This is typically set to

    jn/[email protected]. Each JournalNode will substitute _HOST with its

    own fully qualified hostname at startup. The _HOST placeholder

    allows using the same configuration setting on all JournalNodes.

The

keytab file used by each JournalNode daemon to login as its

    service principal. The principal name is configured with

    dfs.journalnode.kerberos.principal.

The

server principal used by the NameNode for web UI SPNEGO

    authentication when Kerberos security is enabled. This is

    typically set to HTTP/[email protected] The SPNEGO server principal

    begins with the prefix HTTP/ by convention.

    If the value is '*', the web server will attempt to login with

    every principal specified in the keytab file

    dfs.web.authentication.kerberos.keytab.

The

server principal used by the JournalNode HTTP Server for

    SPNEGO authentication when Kerberos security is enabled. This is

    typically set to HTTP/[email protected]. The SPNEGO server principal

    begins with the prefix HTTP/ by convention.

    If the value is '*', the web server will attempt to login with

    every principal specified in the keytab file

    dfs.web.authentication.kerberos.keytab.

    For most deployments this can be set to ${dfs.web.authentication.kerberos.principal}

    i.e use the value of dfs.web.authentication.kerberos.principal.

The

server principal used by the Secondary NameNode for web UI SPNEGO

    authentication when Kerberos security is enabled. Like all other

    Secondary NameNode settings, it is ignored in an HA setup.

    If the value is '*', the web server will attempt to login with

    every principal specified in the keytab file

    dfs.web.authentication.kerberos.keytab.

The

server principal used by the NameNode for WebHDFS SPNEGO

    authentication.

    Required when WebHDFS and security are enabled. In most secure clusters this

    setting is also used to specify the values for

    dfs.namenode.kerberos.internal.spnego.principal and

    dfs.journalnode.kerberos.internal.spnego.principal.

The

keytab file for the principal corresponding to

    dfs.web.authentication.kerberos.principal.

A

client-side RegEx that can be configured to control

    allowed realms to authenticate with (useful in cross-realm env.)

Indicate

whether or not to avoid reading from "stale" datanodes whose

    heartbeat messages have not been received by the namenode

    for more than a specified time interval. Stale datanodes will be

    moved to the end of the node list returned for reading. See

    dfs.namenode.avoid.write.stale.datanode for a similar setting for writes.

Indicate

whether or not to avoid writing to "stale" datanodes whose

    heartbeat messages have not been received by the namenode

    for more than a specified time interval. Writes will avoid using

    stale datanodes unless more than a configured ratio

    (dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as

    stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting

    for reads.

Default

time interval in milliseconds for marking a datanode as "stale",

    i.e., if the namenode has not received heartbeat msg from a datanode for

    more than this time interval, the datanode will be marked and treated

    as "stale" by default. The stale interval cannot be too small since

    otherwise this may cause too frequent change of stale states.

    We thus set a minimum stale interval value (the default value is 3 times

    of heartbeat interval) and guarantee that the stale interval cannot be less

    than the minimum value. A stale data node is avoided during lease/block

    recovery. It can be conditionally avoided for reads (see

    dfs.namenode.avoid.read.stale.datanode) and for writes (see

    dfs.namenode.avoid.write.stale.datanode).

When

the ratio of number stale datanodes to total datanodes marked

    is greater than this ratio, stop avoiding writing to stale nodes so

    as to prevent causing hotspots.

*Note*:

Advanced property. Change with caution.

    This determines the percentage amount of block

    invalidations (deletes) to do over a single DN heartbeat

    deletion command. The final deletion count is determined by applying this

    percentage to the number of live nodes in the system.

    The resultant number is the number of blocks from the deletion list

    chosen for proper invalidation over a single heartbeat of a single DN.

    Value should be a positive, non-zero percentage in float notation (X.Yf),

    with 1.0f meaning 100%.

*Note*:

Advanced property. Change with caution.

    This determines the total amount of block transfers to begin in

    parallel at a DN, for replication, when such a command list is being

    sent over a DN heartbeat by the NN. The actual number is obtained by

    multiplying this multiplier with the total number of live nodes in the

    cluster. The result number is the number of blocks to begin transfers

    immediately for, per DN heartbeat. This number can be any positive,

    non-zero integer.

Specify the port

number used by Hadoop NFS.

Specify the port

number used by Hadoop mount daemon.

This

directory is used to temporarily save out-of-order writes before

    writing to HDFS. For each file, the out-of-order writes are dumped after

    they are accumulated to exceed certain threshold (e.g., 1MB) in memory.

    One needs to make sure the directory has enough space.

This

is the maximum size in bytes of a READ request

    supported by the NFS gateway. If you change this, make sure you

    also update the nfs mount's rsize(add rsize= # of bytes to the

    mount directive).

This

is the maximum size in bytes of a WRITE request

    supported by the NFS gateway. If you change this, make sure you

    also update the nfs mount's wsize(add wsize= # of bytes to the

    mount directive).

*Note*:

Advanced property. Change with caution.

    This is the path to the keytab file for the hdfs-nfs gateway.

    This is required when the cluster is kerberized.

*Note*:

Advanced property. Change with caution.

    This is the name of the kerberos principal. This is required when

    the cluster is kerberized.It must be of this format:

    nfs-gateway-user/nfs-gateway-host@kerberos-realm

When

set to false, client connections originating from unprivileged ports

    (those above 1023) will be rejected. This is to ensure that clients

    connecting to this NFS Gateway must have had root privilege on the machine

    where they're connecting from.

The

minimum number of seconds that we'll cache libhdfs connection objects

    in fuse_dfs. Lower values will result in lower memory consumption; higher

    values may speed up access by avoiding the overhead of creating new

    connection objects.

The

number of seconds between cache expiry checks in fuse_dfs. Lower values

    will result in fuse_dfs noticing changes to Kerberos ticket caches more

    quickly.

This

setting controls how frequently the NameNode logs its metrics. The

    logging configuration must also define one or more appenders for

    NameNodeMetricsLog for the metrics to be logged.

    NameNode metrics logging is disabled if this value is set to zero or

    less than zero.

This

setting controls how frequently the DataNode logs its metrics. The

    logging configuration must also define one or more appenders for

    DataNodeMetricsLog for the metrics to be logged.

    DataNode metrics logging is disabled if this value is set to zero or

    less than zero.

Comma-delimited

set of integers denoting the desired rollover intervals

    (in seconds) for percentile latency metrics on the Namenode and Datanode.

    By default, percentile latency metrics are disabled.

A switch to turn

on/off tracking DataNode peer statistics.

This

setting controls how frequently DataNodes will report their peer

    latencies to the NameNode via heartbeats.  This setting supports

    multiple time unit suffixes as described in dfs.heartbeat.interval.

    If no suffix is specified then milliseconds is assumed.

    It is ignored if dfs.datanode.peer.stats.enabled is false.

This

setting controls the percentage of file I/O events which will be

    profiled for DataNode disk statistics. The default value of 0 disables

    disk statistics. Set to an integer value between 1 and 100 to enable disk

    statistics.

A

comma-separated list of the granularity in seconds for the metrics

    which describe the 50/75/90/95/99th percentile latency for group resolution

    in milliseconds.

    By default, percentile latency metrics are disabled.

Whether

or not actual block data that is read/written from/to HDFS should

    be encrypted on the wire. This only needs to be set on the NN and DNs,

    clients will deduce this automatically. It is possible to override this setting

    per connection by specifying custom logic via dfs.trustedchannel.resolver.class.

This

value may be set to either "3des" or "rc4". If nothing is set, then

    the configured JCE default on the system is used (usually 3DES.) It is

    widely believed that 3DES is more cryptographically secure, but RC4 is

    substantially faster.

    Note that if AES is supported by both the client and server then this

    encryption algorithm will only be used to initially transfer keys for AES.

    (See dfs.encrypt.data.transfer.cipher.suites.)

This

value may be either undefined or AES/CTR/NoPadding.  If defined, then

    dfs.encrypt.data.transfer uses the specified cipher suite for data

    encryption.  If not defined, then only the algorithm specified in

    dfs.encrypt.data.transfer.algorithm is used.  By default, the property is

    not defined.

The

key bitlength negotiated by dfsclient and datanode for encryption.

    This value may be set to either 128, 192 or 256.

TrustedChannelResolver

is used to determine whether a channel

      is trusted for plain data transfer. The TrustedChannelResolver is

      invoked on both client and server side. If the resolver indicates

      that the channel is trusted, then the data transfer will not be

      encrypted even if dfs.encrypt.data.transfer is set to true. The

      default implementation returns false indicating that the channel

      is not trusted.

A

comma-separated list of SASL protection values used for secured

    connections to the DataNode when reading or writing block data.  Possible

    values are authentication, integrity and privacy.  authentication means

    authentication only and no integrity or privacy; integrity implies

    authentication and integrity are enabled; and privacy implies all of

    authentication, integrity and privacy are enabled.  If

    dfs.encrypt.data.transfer is set to true, then it supersedes the setting for

    dfs.data.transfer.protection and enforces that all connections must use a

    specialized encrypted SASL handshake.  This property is ignored for

    connections to a DataNode listening on a privileged port.  In this case, it

    is assumed that the use of a privileged port establishes sufficient trust.

SaslPropertiesResolver

used to resolve the QOP used for a connection to the

    DataNode when reading or writing block data. If not specified, the value of

    hadoop.security.saslproperties.resolver.class is used as the default value.

The JournalNode RPC

server address and port.

The

address and port the JournalNode HTTP server listens on.

    If the port is 0 then the server will start on a free port.

The

address and port the JournalNode HTTPS server listens on.

    If the port is 0 then the server will start on a free port.

List

of classes implementing audit loggers that will receive audit events.

    These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger.

    The special value "default" can be used to reference the default audit

    logger, which uses the configured log system. Installing custom audit loggers

    may affect the performance and stability of the NameNode. Refer to the custom

    logger's documentation for more details.

Only

used when the dfs.datanode.fsdataset.volume.choosing.policy is set to

    org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.

    This setting controls how much DN volumes are allowed to differ in terms of

    bytes of free disk space before they are considered imbalanced. If the free

    space of all the volumes are within this range of each other, the volumes

    will be considered balanced and block assignments will be done on a pure

    round robin basis.

Only

used when the dfs.datanode.fsdataset.volume.choosing.policy is set to

    org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.

    This setting controls what percentage of new block allocations will be sent

    to volumes with more available disk space than others. This setting should

    be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should

    be no reason to prefer that volumes with less available disk space receive

    more block allocations.

Specifies

whether to flush edit log file channel. When set, expensive

    FileChannel#force calls are skipped and synchronous disk writes are

    enabled instead by opening the edit log file with RandomAccessFile("rws")

    flags. This can significantly improve the performance of edit log writes

    on the Windows platform.

    Note that the behavior of the "rws" flags is platform and hardware specific

    and might not provide the same level of guarantees as FileChannel#force.

    For example, the write will skip the disk-cache on SAS and SCSI devices

    while it might not on SATA devices. This is an expert level setting,

    change with caution.

Just

like dfs.datanode.drop.cache.behind.writes, this setting causes the

    page cache to be dropped behind HDFS writes, potentially freeing up more

    memory for other uses.  Unlike dfs.datanode.drop.cache.behind.writes, this

    is a client-side setting rather than a setting for the entire datanode.

    If present, this setting will override the DataNode default.

    If the native libraries are not available to the DataNode, this

    configuration has no effect.

Just

like dfs.datanode.drop.cache.behind.reads, this setting causes the

    page cache to be dropped behind HDFS reads, potentially freeing up more

    memory for other uses.  Unlike dfs.datanode.drop.cache.behind.reads, this

    is a client-side setting rather than a setting for the entire datanode.  If

    present, this setting will override the DataNode default.

    If the native libraries are not available to the DataNode, this

    configuration has no effect.

When

using remote reads, this setting causes the datanode to

    read ahead in the block file using posix_fadvise, potentially decreasing

    I/O wait times.  Unlike dfs.datanode.readahead.bytes, this is a client-side

    setting rather than a setting for the entire datanode.  If present, this

    setting will override the DataNode default.

    When using local reads, this setting determines how much readahead we do in

    BlockReaderLocal.

    If the native libraries are not available to the DataNode, this

    configuration has no effect.

The

amount of milliseconds after which cached server defaults are updated.

    By default this parameter is set to 1 hour.

This

enables the retry cache on the namenode. Namenode tracks for

    non-idempotent requests the corresponding response. If a client retries the

    request, the response from the retry cache is sent. Such operations

    are tagged with annotation @AtMostOnce in namenode protocols. It is

    recommended that this flag be set to true. Setting it to false, will result

    in clients getting failure responses to retried request. This flag must

    be enabled in HA setup for transparent fail-overs.

    The entries in the cache have expiration time configurable

    using dfs.namenode.retrycache.expirytime.millis.

The time for which

retry cache entries are retained.

This

parameter configures the heap size allocated for retry cache

    (excluding the response cached). This corresponds to approximately

    4096 entries for every 64MB of namenode process java heap size.

    Assuming retry cache entry expiration time (configured using

    dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this

    enables retry cache to support 7 operations per second sustained

    for 10 minutes. As the heap size is increased, the operation rate

    linearly increases.

If this is set to

false, the client won't attempt to perform memory-mapped reads.

When

zero-copy reads are used, the DFSClient keeps a cache of recently used

    memory mapped regions.  This parameter controls the maximum number of

    entries that we will keep in that cache.

    The larger this number is, the more file descriptors we will potentially

    use for memory-mapped files.  mmaped files also use virtual address space.

    You may need to increase your ulimit virtual address space limits before

    increasing the client mmap cache size.

    Note that you can still do zero-copy reads when this size is set to 0.

The

minimum length of time that we will keep an mmap entry in the cache

    between uses.  If an entry is in the cache longer than this, and nobody

    uses it, it will be removed by a background thread.

The

minimum amount of time that we will wait before retrying a failed mmap

    operation.

The

maximum amount of time that we will consider a short-circuit replica to

    be valid, if there is no communication from the DataNode.  After this time

    has elapsed, we will re-fetch the short-circuit replica even if it is in

    the cache.

The

percentage of the Java heap which we will allocate to the cached blocks

    map.  The cached blocks map is a hash map which uses chained hashing.

    Smaller maps may be accessed more slowly if the number of cached blocks is

    large; larger maps will consume more memory.

The

amount of memory in bytes to use for caching of block replicas in

    memory on the datanode. The datanode's maximum locked memory soft ulimit

    (RLIMIT_MEMLOCK) must be set to at least this value, else the datanode

    will abort on startup.

    By default, this parameter is set to 0, which disables in-memory caching.

    If the native libraries are not available to the DataNode, this

    configuration has no effect.

This

value controls the number of cache directives that the NameNode will

    send over the wire in response to a listDirectives RPC.

This

value controls the number of cache pools that the NameNode will

    send over the wire in response to a listPools RPC.

The

amount of milliseconds between subsequent path cache rescans.  Path

    cache rescans are when we calculate which blocks should be cached, and on

    what datanodes.

    By default, this parameter is set to 30 seconds.

When

the NameNode needs to uncache something that is cached, or cache

    something that is not cached, it must direct the DataNodes to do so by

    sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode

    heartbeat.  This parameter controls how frequently the NameNode will

    resend these commands.

The

maximum number of threads per volume to use for caching new data

    on the datanode. These threads consume both I/O and CPU. This can affect

    normal datanode operations.

Determines

cache reporting interval in milliseconds.  After this amount of

    time, the DataNode sends a full report of its cache state to the NameNode.

    The NameNode uses the cache report to update its map of cached blocks to

    DataNode locations.

    This configuration has no effect if in-memory caching has been disabled by

    setting dfs.datanode.max.locked.memory to 0 (which is the default).

    If the native libraries are not available to the DataNode, this

    configuration has no effect.

Determines

when an active namenode will roll its own edit log.

    The actual threshold (in number of edits) is determined by multiplying

    this value by dfs.namenode.checkpoint.txns.

    This prevents extremely large edit files from accumulating on the active

    namenode, which can cause timeouts during namenode startup and pose an

    administrative hassle. This behavior is intended as a failsafe for when

    the standby or secondary namenode fail to roll the edit log by the normal

    checkpoint threshold.

How

often an active namenode will check if it needs to roll its edit log,

    in milliseconds.

Valid pattern for

user and group names for webhdfs, it must be a valid java regex.

Valid pattern for

user and group names in webhdfs acl operations, it must be a valid java regex.

Socket

timeout for connecting to WebHDFS servers. This prevents a

    WebHDFS client from hanging if the server hostname is

    misconfigured, or the server does not response before the timeout

    expires. Value is followed by a unit specifier: ns, us, ms, s, m,

    h, d for nanoseconds, microseconds, milliseconds, seconds,

    minutes, hours, days respectively. Values should provide units,

    but milliseconds are assumed.

Socket

timeout for reading data from WebHDFS servers. This

    prevents a WebHDFS client from hanging if the server stops sending

    data. Value is followed by a unit specifier: ns, us, ms, s, m, h,

    d for nanoseconds, microseconds, milliseconds, seconds, minutes,

    hours, days respectively. Values should provide units,

    but milliseconds are assumed.

The

name of the DFSClient context that we should use.  Clients that share

    a context share a socket cache and short-circuit cache, among other things.

    You should only change this if you don't want to share with another set of

    threads.

This configuration

parameter turns on short-circuit local reads.

Socket

send buffer size for a write pipeline in DFSClient side.

    This may affect TCP connection throughput.

    If it is set to zero or negative value,

    no buffer size will be set explicitly,

    thus enable tcp auto-tuning on some system.

    The default value is 0.

Optional.  This is a path to a UNIX domain socket that

will be used for

    communication between the DataNode and local HDFS clients.

    If the string "_PORT" is present in this path, it will be replaced by the

    TCP port of the DataNode.

If

this configuration parameter is set,

    short-circuit local reads will skip checksums.

    This is normally not recommended,

    but it may be useful for special setups.

    You might consider using this

    if you are doing your own checksumming outside of HDFS.

The

DFSClient maintains a cache of recently opened file descriptors.

    This parameter controls the maximum number of file descriptors in the cache.

    Setting this higher will use more file descriptors,

    but potentially provide better performance on workloads

    involving lots of seeks.

This

controls the minimum amount of time

    file descriptors need to sit in the client cache context

    before they can be closed for being inactive for too long.

Comma

separated paths to the directory on which

    shared memory segments are created.

    The client and the DataNode exchange information via

    this shared memory segment.

    It tries paths in order until creation of shared memory segment succeeds.

A

comma separated list of NameNode commands that are written to the HDFS

    namenode audit log only if the audit log level is debug.

Legacy

short-circuit reader implementation based on HDFS-2246 is used

    if this configuration parameter is true.

    This is for the platforms other than Linux

    where the new implementation based on HDFS-347 is not available.

Comma

separated list of the users allowed to open block files

    on legacy short-circuit local read.

This

control whether we will try to pass normal data traffic

    over UNIX domain socket rather than over TCP socket

    on node-local data transfer.

    This is currently experimental and turned off by default.

If

the value is set to true, then namenode will reject datanode

    registration if the topology mapping for a datanode is not resolved and

    NULL is returned (script defined by net.topology.script.file.name fails

    to execute). Otherwise, datanode will be registered and the default rack

    will be assigned as the topology path. Topology paths are important for

    data resiliency, since they define fault domains. Thus it may be unwanted

    behavior to allow datanode registration with the default rack if the

    resolving topology failed.

Whether support for

extended attributes is enabled on the NameNode.

Maximum number of

extended attributes per inode.

The

maximum combined size of the name and value of an extended attribute

    in bytes. It should be larger than 0, and less than or equal to maximum

    size hard limit which is 32768.

The

threshold in milliseconds at which we will log a slow

    io warning in a dfsclient. By default, this parameter is set to 30000

    milliseconds (30 seconds).

The

threshold in milliseconds at which we will log a slow

    io warning in a datanode. By default, this parameter is set to 300

    milliseconds.

During

the release of lease a lock is hold that make any

    operations on the namenode stuck. In order to not block them during

    a too long duration we stop releasing lease after this max lock limit.

During

the release of lease a lock is hold that make any

    operations on the namenode stuck. In order to not block them during

    a too long duration we stop releasing lease after this max lock limit.

When

a write lock is held on the namenode for a long time,

    this will be logged as the lock is released. This sets how long the

    lock must be held for logging to occur.

When

a read lock is held on the namenode for a long time,

    this will be logged as the lock is released. This sets how long the

    lock must be held for logging to occur.

If

true, the namenode will keep track of how long various

    operations hold the Namesystem lock for and emit this as metrics. These

    metrics have names of the form FSN(Read|Write)LockNanosOperationName,

    where OperationName denotes the name of the operation that initiated the

    lock hold (this will be OTHER for certain uncategorized operations) and

    they export the hold time values in nanoseconds.

If

this is true, the FS Namesystem lock will be used in Fair mode,

    which will help to prevent writer threads from being starved, but can provide

    lower lock throughput. See java.util.concurrent.locks.ReentrantReadWriteLock

    for more information on fair/non-fair locks.

The

delay in seconds at which we will pause the blocks deletion

    after Namenode startup. By default it's disabled.

    In the case a directory has large number of directories and files are

    deleted, suggested delay is one hour to give the administrator enough time

    to notice large number of pending deletion blocks and take corrective

    action.

The

number of threads to use when creating hard links from

    current to previous blocks during upgrade of a DataNode to block ID-based

    block layout (see HDFS-6482 for details on the layout).

When

listing encryption zones, the maximum number of zones

    that will be returned in a batch. Fetching the list incrementally in

    batches improves namenode performance.

When

listing re-encryption status, the maximum number of zones

    that will be returned in a batch. Fetching the list incrementally in

    batches improves namenode performance.

When

listing open files, the maximum number of open files that will be

      returned in a single batch. Fetching the list incrementally in batches

      improves namenode performance.

When

KeyProvider is configured, the interval time of warming

    up edek cache on NN starts up / becomes active. All edeks will be loaded

    from KMS into provider cache. The edek cache loader will try to warm up the

    cache until succeed or NN leaves active state.

When

KeyProvider is configured, the time delayed until the first

    attempt to warm up edek cache on NN start up / become active.

Interval

the re-encrypt EDEK thread sleeps in the main loop. The

    interval accepts units. If none given, millisecond is assumed.

How many EDEKs should

the re-encrypt thread process in one batch.

Throttling

ratio for the re-encryption, indicating what fraction

    of time should the re-encrypt handler thread work under NN read lock.

    Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are

    invalid values and will fail NN startup.

Throttling

ratio for the re-encryption, indicating what fraction

    of time should the re-encrypt updater thread work under NN write lock.

    Larger than 1.0 values are interpreted as 1.0. Negative value or 0 are

    invalid values and will fail NN startup.

Maximum

number of re-encrypt threads to contact the KMS

    and re-encrypt the edeks.

Maximum

number of events that will be sent to an inotify client

    in a single RPC response. The default value attempts to amortize away

    the overhead for this RPC while avoiding huge memory requirements for the

    client and NameNode (1000 events should consume no more than 1 MB.)

The

directory to prepend to user name to get the user's

    home direcotry.

When

the DFSClient reads from a block file which the DataNode is

    caching, the DFSClient can skip verifying checksums.  The DataNode will

    keep the block file in cache until the client is done.  If the client takes

    an unusually long time, though, the DataNode may need to evict the block

    file from the cache anyway.  This value controls how long the DataNode will

    wait for the client to release a replica that it is reading without

    checksums.

How

often the DataNode should poll to see if the clients have

    stopped using a replica that the DataNode wants to uncache.

Allow users to change

the storage policy on files and directories.

Determines

where to save the namespace in the old fsimage format

    during checkpointing by standby NameNode or SecondaryNameNode. Users can

    dump the contents of the old format fsimage by oiv_legacy command. If

    the value is not specified, old format fsimage will not be saved in

    checkpoint.

Enable nntop:

reporting top users on namenode

Number of buckets in

the rolling window implementation of nntop

Number of top users

returned by the top tool

comma separated list

of nntop reporting periods in minutes

How

long in milliseconds after the last access

      the cached UGI will expire. With 0, never expire.

Number

of blocks to rescan for each iteration of

    postponedMisreplicatedBlocks.

Whether pin blocks on

favored DataNode.

The

initial delay (unit is ms) for locateFollowingBlock,

    the delay time will increase exponentially(double) for each retry.

The

HTTP connection and read timeout value (unit is ms ) when DFS ZKFC

    tries to get local NN thread dump after local NN becomes

    SERVICE_NOT_RESPONDING or SERVICE_UNHEALTHY.

    If it is set to zero, DFS ZKFC won't get local NN thread dump.

Whether

enable standby namenode to tail in-progress edit logs.

    Clients might want to turn it on when they want Standby NN to have

    more up-to-date data.

The

default erasure coding policy name will be used

    on the path if no policy name is passed.

The maximum cell size

of erasure coding policy. Default is 4MB.

Datanode striped read

timeout in milliseconds.

Datanode striped read

buffer size.

Number

of threads used by the Datanode for background

    reconstruction work.

Datanode

uses xmits weight to calculate the relative cost of EC recovery

    tasks comparing to replicated block recovery, of which xmits is always 1.

    Namenode then uses xmits reported from datanode to throttle recovery tasks

    for EC and replicated blocks.

    The xmits of an erasure coding recovery task is calculated as the maximum

    value between the number of read streams and the number of write streams.

The

number of concurrent threads to be used in quota initialization. The

    speed of quota initialization also affects the namenode fail-over latency.

    If the size of name space is big, try increasing this.

Socket

send buffer size for DataXceiver (mirroring packets to downstream

    in pipeline). This may affect TCP connection throughput.

    If it is set to zero or negative value, no buffer size will be set

    explicitly, thus enable tcp auto-tuning on some system.

    The default value is 0.

Socket

receive buffer size for DataXceiver (receiving packets from client

    during block writing). This may affect TCP connection throughput.

    If it is set to zero or negative value, no buffer size will be set

    explicitly, thus enable tcp auto-tuning on some system.

    The default value is 0.

This

is valid only when block placement policy is set to

    BlockPlacementPolicyWithUpgradeDomain. It defines the number of

    unique upgrade domains any block's replicas should have.

    When the number of replicas is less or equal to this value, the policy

    ensures each replica has an unique upgrade domain. When the number of

    replicas is greater than this value, the policy ensures the number of

    unique domains is at least this value.

RPC port for

Zookeeper Failover Controller.

The

maximum wait time for datanode to be ready before failing the

    received request. Setting this to 0 fails requests right away if the

    datanode is not yet registered with the namenode. This wait time

    reduces initial request failures after datanode restart.

    Support multiple time unit suffix(case insensitive), as described

    in dfs.heartbeat.interval.

The

interval check time of loading DU_CACHE_FILE in each volume.

    When the cluster doing the rolling upgrade operations, it will

    usually lead dfsUsed cache file of each volume expired and redo the

    du operations in datanode and that makes datanode start slowly. Adjust

    this property can make cache file be available for the time as you want.

If

true, then enables WebHDFS protection against cross-site request forgery

    (CSRF).  The WebHDFS client also uses this property to determine whether or

    not it needs to send the custom CSRF prevention header in its HTTP requests.

The

name of a custom header that HTTP requests must send when protection

    against cross-site request forgery (CSRF) is enabled for WebHDFS by setting

    dfs.webhdfs.rest-csrf.enabled to true.  The WebHDFS client also uses this

    property to determine whether or not it needs to send the custom CSRF

    prevention header in its HTTP requests.

A

comma-separated list of HTTP methods that do not require HTTP requests to

    include a custom header when protection against cross-site request forgery

    (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to

    true.  The WebHDFS client also uses this property to determine whether or

    not it needs to send the custom CSRF prevention header in its HTTP requests.

A

comma-separated list of regular expressions used to match against an HTTP

    request's User-Agent header when protection against cross-site request

    forgery (CSRF) is enabled for WebHDFS by setting

    dfs.webhdfs.reset-csrf.enabled to true.  If the incoming User-Agent matches

    any of these regular expressions, then the request is considered to be sent

    by a browser, and therefore CSRF prevention is enforced.  If the request's

    User-Agent does not match any of these regular expressions, then the request

    is considered to be sent by something other than a browser, such as scripted

    automation.  In this case, CSRF is not a potential attack vector, so

    the prevention is not enforced.  This helps achieve backwards-compatibility

    with existing automation that has not been updated to send the CSRF

    prevention header.

If

true, then enables protection against clickjacking by returning

      X_FRAME_OPTIONS header value set to SAMEORIGIN.

      Clickjacking protection prevents an attacker from using transparent or

      opaque layers to trick a user into clicking on a button

      or link on another page.

This

configration value allows user to specify the value for the

      X-FRAME-OPTIONS. The possible values for this field are

      DENY, SAMEORIGIN and ALLOW-FROM. Any other value will throw an

      exception when namenode and datanodes are starting up.

Set to true to enable

login using a keytab for Kerberized Hadoop.

The

hostname used for a keytab based Kerberos login. Keytab based login

    can be enabled with dfs.balancer.keytab.enabled.

The

keytab file used by the Balancer to login as its

    service principal. The principal name is configured with

    dfs.balancer.kerberos.principal. Keytab based login can be

    enabled with dfs.balancer.keytab.enabled.

The

Balancer principal. This is typically set to

    balancer/[email protected]. The Balancer will substitute _HOST with its

    own fully qualified hostname at startup. The _HOST placeholder

    allows using the same configuration setting on different servers.

    Keytab based login can be enabled with dfs.balancer.keytab.enabled.

If

"true", enable the retry policy of WebHDFS client.

    If "false", retry policy is turned off.

    Enabling the retry policy can be quite useful while using WebHDFS to

    copy large files between clusters that could timeout, or

    copy files between HA clusters that could failover during the copy.

Specify

a policy of multiple linear random retry for WebHDFS client,

    e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1),

    ..., the first n0 retries sleep t0 milliseconds on average,

    the following n1 retries sleep t1 milliseconds on average, and so on.

Specify

the max number of failover attempts for WebHDFS client

    in case of network exception.

Specify

the max number of retry attempts for WebHDFS client,

    if the difference between retried attempts and failovered attempts is

    larger than the max number of retry attempts, there will be no more

    retries.

Specify

the base amount of time in milliseconds upon which the

    exponentially increased sleep time between retries or failovers

    is calculated for WebHDFS client.

Specify

the upper bound of sleep time in milliseconds between

    retries or failovers for WebHDFS client.

The

class that provides access for host files.

    org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager is used

    by default which loads files specified by dfs.hosts and dfs.hosts.exclude.

    If org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager is

    used, it will load the JSON file defined in dfs.hosts.

    To change class name, nn restart is required. "dfsadmin -refreshNodes" only

    refreshes the configuration files used by the class.

HTTPS port for

DataNode.

Size

of the thread pool for the HDFS balancer block mover.

    dispatchExecutor

Window

of time in ms for the HDFS balancer tracking blocks and its

    locations.

Thread

pool size for executing block moves.

    moverThreadAllocator

Maximum

number of bytes that can be moved by the balancer in a single

    thread.

Minimum

block threshold size in bytes to ignore when fetching a source's

    block list.

Total

size in bytes of Datanode blocks to get when fetching a source's

    block list.

Maximum

amount of time in milliseconds for a block to move. If this is set

    greater than 0, Balancer will stop waiting for a block move completion

    after this time. In typical clusters, a 3 to 5 minute timeout is reasonable.

    If timeout happens to a large proportion of block moves, this needs to be

    increased. It could also be that too much work is dispatched and many nodes

    are constantly exceeding the bandwidth limit as a result. In that case,

    other balancer parameters might need to be adjusted.

    It is disabled (0) by default.

If

this specified amount of time has elapsed and no block has been moved

    out of a source DataNode, on more effort will be made to move blocks out of

    this DataNode in the current Balancer iteration.

The

maximum number of invalidate blocks sent by namenode to a datanode

    per heartbeat deletion command. This property works with

    "dfs.namenode.invalidate.work.pct.per.iteration" to throttle block

    deletions.

Maximum number of

blocks to process for initializing replication queues.

Placement

policy class for striped files.

    Defaults to BlockPlacementPolicyRackFaultTolerant.class

Class

representing block placement policy for non-striped files.

    There are four block placement policies currently being supported:

    BlockPlacementPolicyDefault, BlockPlacementPolicyWithNodeGroup,

    BlockPlacementPolicyRackFaultTolerant and BlockPlacementPolicyWithUpgradeDomain.

    BlockPlacementPolicyDefault chooses the desired number of targets

    for placing block replicas in a default way. BlockPlacementPolicyWithNodeGroup

    places block replicas on environment with node-group layer. BlockPlacementPolicyRackFaultTolerant

    places the replicas to more racks.

    BlockPlacementPolicyWithUpgradeDomain places block replicas that honors upgrade domain policy.

    The details of placing replicas are documented in the javadoc of the corresponding policy classes.

    The default policy is BlockPlacementPolicyDefault, and the corresponding class is

    org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.

If

set to a positive integer, the value in ms to wait between sending

    incremental block reports from the Datanode to the Namenode.

Checksum type

Number of retries to

use when finding the next block during HDFS writes.

The

prefix (plus a required nameservice ID) for the class name of the

    configured Failover proxy provider for the host.  For more detailed

    information, please consult the "Configuration Details" section of

    the HDFS High Availability documentation.

DFS client security

key cache expiration in milliseconds.

Maximum failures

allowed when trying to get block information from a specific datanode.

The

number of bytes for the DFSClient will fetch from the Namenode

    during a read operation.  Defaults to 10 * ${dfs.blocksize}.

Threshold in

milliseconds for read entries during short-circuit local reads.

Buffer size in bytes

for short-circuit local reads.

The

maximum number of threads used for parallel reading

    in striped layout.

Comma-separated

classes for building ReplicaAccessor.  If the classes

    are specified, client will use external BlockReader that uses the

    ReplicaAccessor built by the builder.

Retry

interval in milliseconds to wait between retries in getting

    block lengths from the datanodes.

Max retry attempts

for DFSClient talking to namenodes.

If true, turns on

DFSClient retry policy.

Set to pairs of

timeouts and retries for DFSClient.

Number of retries for

calls to fetchLocatedBlocksAndGetLastBlockLength().

Base

time window in ms for DFSClient retries.  For each retry attempt,

    this value is extended linearly (e.g. 3000 ms for first attempt and

    first retry, 6000 ms for second retry, 9000 ms for third retry, etc.).

Default timeout value

in milliseconds for all sockets.

Socket cache capacity

(in entries) for short-circuit reads.

Socket cache

expiration for short-circuit reads in msec.

The

number of Namenode responses dropped by DFSClient for each RPC call.  Used

    for testing the NN retry cache.

Support

'hedged' reads in DFSClient. To enable this feature, set the parameter

    to a positive number. The threadpool size is how many threads to dedicate

    to the running of these 'hedged', concurrent reads in your client.

Configure

'hedged' reads in DFSClient. This is the number of milliseconds

    to wait before starting up a 'hedged' read.

The maximum number of

arrays allowed for each array length.

The

time period in milliseconds that the allocation count for each array length is

    reset to zero if there is no increment.

The

count threshold for each array length so that a manager is created only after the

    allocation count exceeds the threshold. In other words, the particular array length

    is not managed until the allocation count exceeds the threshold.

If true, enables byte

array manager used by DFSOutputStream.

The maximum number of

DFSPackets allowed in flight.

The

maximum content summary counts allowed in one locking period. 0 or a negative number

    means no limit (i.e. no yielding).

The

length of time in microseconds to put the thread to sleep, between reaquiring the locks

    in content summary computation.

If true, set

TCP_NODELAY to sockets for transferring data from DFS client.

If true, set

TCP_NODELAY to sockets for transferring data between Datanodes.

Maximum

number of threads for Datanode balancer pending moves.  This

    value is reconfigurable via the "dfsadmin -reconfig" command.

The

class name for the underlying storage that stores replicas for a

    Datanode.  Defaults to

    org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.

The

class name of the policy for choosing volumes in the list of

    directories.  Defaults to

    org.apache.hadoop.hdfs.server.datanode.fsdataset.RoundRobinVolumeChoosingPolicy.

    If you would like to take into account available disk space, set the

    value to

    "org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy".

Optional.  The hostname for the Datanode containing

this

    configuration file.  Will be different for each machine.

    Defaults to current hostname.

Interval in seconds

for Datanodes for lazy persist writes.

The

maximum number of entries the datanode per-host network error

    count cache may contain.

Timeout

value when sending OOB response for each OOB type, which are

    OOB_RESTART, OOB_RESERVED1, OOB_RESERVED2, and OOB_RESERVED3,

    respectively.  Currently, only OOB_RESTART is used.

Maximum

number of threads to use for upgrading data directories.

    The default value is the number of storage directories in the

    DataNode.

Name

of the class implementing the RamDiskReplicaTracker interface.

    Defaults to

    org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaLruTracker.

During

shutdown for restart, the amount of time in seconds budgeted for

    datanode restart.

The

window of time in ms before the DataXceiver closes a socket for a

    single request.  If a second request occurs within that window, the

    socket can be reused.

Timeout in ms for

clients socket writes to DataNodes.

If

set to true, then sync_file_range() system call will occur

    asynchronously.  This property is only valid when the property

    dfs.datanode.sync.behind.writes is true.

If

false, break block transfers on 32-bit machines greater than

    or equal to 2GB into smaller chunks.

A

list of scripts or Java classes which will be used to fence

    the Active NameNode during a failover.  See the HDFS High

    Availability documentation for details on automatic HA

    configuration.

If

true, a NameNode in Standby state periodically takes a checkpoint

    of the namespace, saves it to its local storage and then upload to

    the remote NameNode.

The

port number that the zookeeper failover controller RPC

    server binds to.

The directory where

the journal edit files are stored.

If

true, the journal nodes wil sync with each other. The journal nodes

    will periodically gossip with other journal nodes to compare edit log

    manifests and if they detect any missing log segment, they will download

    it from the other journal nodes.

Time

interval, in milliseconds, between two Journal Node syncs.

    This configuration takes effect only if the journalnode sync is enabled

    by setting the configuration parameter dfs.journalnode.enable.sync to true.

Kerberos SPNEGO

principal name used by the journal node.

Kerberos principal

name for the journal node.

Kerberos keytab file

for the journal node.

Limit

the number of files printed by ls. If less or equal to

    zero, at most DFS_LIST_LIMIT_DEFAULT (= 1000) will be printed.

The

minimum time interval, in milliseconds, that a block can be

    moved to another location again.

Configure the

balancer's mover thread pool size.

The

maximum number of retries before the mover consider the

    move failed.

Set to true to enable

login using a keytab for Kerberized Hadoop.

The

hostname used for a keytab based Kerberos login. Keytab based login

    can be enabled with dfs.mover.keytab.enabled.

The

keytab file used by the Mover to login as its

    service principal. The principal name is configured with

    dfs.mover.kerberos.principal. Keytab based login can be

    enabled with dfs.mover.keytab.enabled.

The

Mover principal. This is typically set to

    mover/[email protected]. The Mover will substitute _HOST with its

    own fully qualified hostname at startup. The _HOST placeholder

    allows using the same configuration setting on different servers.

    Keytab based login can be enabled with dfs.mover.keytab.enabled.

If

this specified amount of time has elapsed and no block has been moved

    out of a source DataNode, on more effort will be made to move blocks out of

    this DataNode in the current Mover iteration.

If true, enables

asynchronous audit log.

If true, adds a

tracking ID for all audit log events.

Only

used when the dfs.block.replicator.classname is set to

    org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.

    Special value between 0 and 1, noninclusive.  Increases chance of

    placing blocks on Datanodes with less disk space used.

Service RPC address

for the backup Namenode.

For

testing.  Setting to true always allows the DT secret manager

    to be used, even if security is disabled.

If

set to true, enables asynchronous edit logs in the Namenode.  If set

    to false, the Namenode uses the traditional synchronous edit logs.

dfs.namenode.edits.dir

includes both required directories

    (specified by dfs.namenode.edits.dir.required) and optional directories.

    The number of usable optional directories must be greater than or equal

    to this property.  If the number of usable optional directories falls

    below dfs.namenode.edits.dir.minimum, HDFS will issue an error.

    This property defaults to 1.

When

FSEditLog is creating JournalManagers from dfs.namenode.edits.dir,

    and it encounters a URI with a schema different to "file" it loads the

    name of the implementing class from

    "dfs.namenode.edits.journal-plugin.[schema]". This class must implement

    JournalManager and have a constructor which takes (Configuration, URI).

Normally

a file can only be closed with all its blocks are committed.

    When this value is set to a positive integer N, a file can be closed

    when N blocks are committed and the rest complete.

Name of class to use

for delegating HDFS authorization.

A

list of user principals (in secure cluster) or user names (in insecure

    cluster) for whom the external attributes provider will be bypassed for all

    operations. This means file attributes stored in HDFS instead of the

    external provider will be used for permission checking and be returned when

    requested.

Puts

a limit on the number of blocks printed to the log by the Namenode

    after a block report.

Maximum opcode size

in bytes.

The

number of checkpoint period windows (as defined by the property

    dfs.namenode.checkpoint.period) allowed by the Namenode to perform

    saving the namespace before shutdown.

Frequently

accessed files that are accessed more times than this

    threshold are cached in the FSDirectory nameCache.

Hard limit for the

number of highest-priority replication streams.

Hard limit for all

replication streams.

Timeout

in seconds for block reconstruction.  If this value is 0 or less,

    then it will default to 5 minutes.

Minimum

number of missed heartbeats intervals for a datanode to

    be marked stale by the Namenode.  The actual interval is calculated as

    (dfs.namenode.stale.datanode.minimum.interval * dfs.heartbeat.interval)

    in seconds.  If this value is greater than the property

    dfs.namenode.stale.datanode.interval, then the calculated value above

    is used.

Timeout value in ms

for the StorageInfo compaction run.

The

thread for checking the StorageInfo for defragmentation will

    run periodically.  The time between runs is determined by this

    property.

The defragmentation

threshold for the StorageInfo.

If

true, snapshots taken will have an immutable shared copy of

    the open files that have valid leases. Even after the open files

    grow or shrink in size, snapshot will always have the previous

    point-in-time version of the open files, just like all other

    closed files. Default is false.

    Note: The file length captured for open files in snapshot is

    whats recorded in NameNode at the time of snapshot and it may

    be shorter than what the client has written till then. In order

    to capture the latest length, the client can call hflush/hsync

    with the flag SyncFlag.UPDATE_LENGTH on the open files handles.

If

accessTime of a file/directory changed but there is no other

    modification made to the file/directory, the changed accesstime will

    not be captured in next snapshot. However, if there is other modification

    made to the file/directory, the latest access time will be captured

    together with the modification in next snapshot.

If

enabled, snapshotDiff command can be run for any descendant directory

    under a snapshot root directory and the diff calculation will be scoped

    to the given descendant directory. Otherwise, snapshot diff command can

    only be run for a snapshot root directory.

If

true, allows ECN (explicit congestion notification) from the

    Datanode.

Quorum

timeout in milliseconds during accept phase of

    recovery/synchronization for a specific segment.

Quorum

timeout in milliseconds during finalizing for a specific

    segment.

Timeout

in milliseconds when calling getJournalState().

    JournalNodes.

Timeout

in milliseconds when getting an epoch number for write

    access to JournalNodes.

Quorum

timeout in milliseconds during preparation phase of

    recovery/synchronization for a specific segment.

Queue size in MB for

quorum journal edits.

Timeout in

milliseconds for accepting streams from JournalManagers.

Quorum timeout in

milliseconds for starting a log segment.

Write

timeout in milliseconds when writing to a quorum of remote

    journals.

If true, enables

quotas based on storage type.

Kerberos principal

name for the Secondary NameNode.

Kerberos keytab file

for the Secondary NameNode.

Authentication filter

class used for WebHDFS.

If

true, allow anonymous user to access WebHDFS. Set to

    false to disable anonymous authentication.

dfs.web.ugi is

deprecated. Use hadoop.http.staticuser.user instead.

High watermark

configuration to Netty for Datanode WebHdfs.

Low watermark

configuration to Netty for Datanode WebHdfs.

Access

token provider class for WebHDFS using OAuth2.

    Defaults to org.apache.hadoop.hdfs.web.oauth2.ConfCredentialBasedAccessTokenProvider.

Client

id used to obtain access token with either credential or

    refresh token.

If true, enables

OAuth2 in WebHDFS

URL

against which to post for obtaining bearer token with

    either credential or refresh token.

Keystore key password

for HTTPS SSL configuration

Keystore location for

HTTPS SSL configuration

Keystore password for

HTTPS SSL configuration

Truststore location

for HTTPS SSL configuration

Truststore password

for HTTPS SSL configuration

Maximum

disk bandwidth used by diskbalancer

      during read from a source disk. The unit is MB/sec.

When

a disk balancer copy operation is proceeding, the datanode is still

      active. So it might not be possible to move the exactly specified

      amount of data. So tolerance allows us to define a percentage which

      defines a good enough move.

During

a block move from a source to destination disk, we might

      encounter various errors. This defines how many errors we can tolerate

      before we declare a move between 2 disks (or a step) has failed.

This

enables the diskbalancer feature on a cluster. By default, disk

      balancer is disabled.

The

percentage threshold value for volume Data Density in a plan.

      If the absolute value of volume Data Density which is out of

      threshold value in a node, it means that the volumes corresponding to

      the disks should do the balancing in the plan. The default value is 10.

Instrumentation

reporting long critical sections will suppress

      consecutive warnings within this interval.

The size buffer to be

used when creating or opening httpfs filesystem IO stream.

Enables

routing of webhdfs calls through rpc

      call queue

The

minimum gap between two successive checks of the same DataNode

      volume. This setting supports multiple time unit suffixes as described

      in dfs.heartbeat.interval. If no suffix is specified then milliseconds

      is assumed.

Maximum

allowed time for a disk check to complete during DataNode

      startup. If the check does not complete within this time interval

      then the disk is declared as failed. This setting supports

      multiple time unit suffixes as described in dfs.heartbeat.interval.

      If no suffix is specified then milliseconds is assumed.

Enables

DFSNetworkTopology to choose nodes for placing replicas.

Common

key to set timeout for related operations in

      QuorumJournalManager. This setting supports multiple time unit suffixes

      as described in dfs.heartbeat.interval.

      If no suffix is specified then milliseconds is assumed.

Disable

reformat of NameNode. If it's value is set to "true"

      and metadata directories already exist then attempt to format NameNode

      will throw NameNodeFormatException.

Nameservice

identifier of the default subcluster to monitor.

If

true, the RPC service to handle client requests in the router is

      enabled.

RPC

address that handles all clients requests.

      The value of this property will take the form of router-host1:rpc-port.

The

actual address the RPC server will bind to. If this optional address is

      set, it overrides only the hostname portion of

      dfs.federation.router.rpc-address. This is useful for making the name node

      listen on all interfaces by setting it to 0.0.0.0.

The

number of server threads for the router to handle RPC requests from

      clients.

The size of the queue

for the number of handlers to handle RPC client requests.

The number of readers

for the router to handle RPC client requests.

The size of the queue

for the number of readers for the router to handle RPC client requests.

Size of the pool of

connections from the router to namenodes.

Time

interval, in milliseconds, to check if the connection pool should

      remove unused connections.

Time

interval, in milliseconds, to check if the connection manager should

      remove unused connection pools.