2015年11月8日 星期日

複製 (Clone) Linux 系統 (包括 Gentoo, CentOS)

完整複製 (clone) 整個檔案系統

下面是使用 cpio 完整複製檔案系統的指引

[引用原來的文章]

HOWTO: Using cpio to Copy a Partition in Linux

Using cpio to copy everything from one partition to another in Linux, maintaining all file permissions, symlinks, and timestamps.

# File: copy_partition.notes
# Auth: burly
# Date: 03/20/05
# Desc: A method of copying everything from one
#      partition to another
 
# Change directories to the top level of the
# partition you want to copy
cd /part
 
# Use find to locate all objects and cpio to copy them
find . -mount -print0 | cpio -0dump /path/to/new/part

How to move your filesystem to a new hard drive

ref: http://www.greenfly.org/tips/filesystem_migration.html

Despite the best planning, it eventually happens. You start to run out of space on your linux system. While there are many different ways to copy files from one hard drive to another, some work better than others when transferring the full / directory. Below I will outline a method I have used about 50 times to transfer full systems and partitions from one machine to another.

Why this can be complicated

When you are copying a full linux system from one partition to another, there are a few issues you need to consider:

  • Preserving permissions - If your files aren't owned by the same people, and with the same permissions, your system won't run as expected to say the least.
  • Properly handling special files - Certain methods of copying a system don't properly handle the /dev and /proc filesystems, as a result, you will boot on the new drive only to find you have no device entries listed.
  • Spanning filesystems - When you are copying one filesystem to another, especially the root filesystem, you don't want to span across filesystems. That is, say you have the new hard drive mounted at /mnt/temp, if you recursively copy / to /mnt/temp and allow spanning filesystems, you could end up in a situation where /mnt/temp is copied into /mnt/temp/mnt/temp and /mnt/temp/mnt/temp is copied into /mnt/temp/mnt/temp/mnt/temp (not to mention the rest of the filesystem you have copied into /mnt/temp). To avoid this, most copying programs have some sort of option to only copy the mounted filesystem it is started from, and not to continue onto other filesystems.

What to do

With all these in mind, the best method I've found for copying the / filesystem is as follows (assuming for the example the new drive is mounted at /mnt/temp):

cd /; find ./ -xdev -print0 | cpio -pa0V /mnt/temp

注意,一定要到 source 的目錄下,才能成功執行

到 source 的目錄下,再執行備份指令
find ./ -xdev -print0 | cpio -pa0V /mnt/temp
產生的 cpio 指令為
cpio: /mnt/temp/./usr/share/icons/Adwaita/24x24/actions/mail-mark-unread.png

假如使用下面的指令
find / -xdev -print0 | cpio -pa0V /mnt/hdd
執行 cpio 指令就會失敗,錯誤訊息如下
cpio: /mnt/hdd not created: newer or same age version exists
When you run this command, it will recursively copy everything on the / filesystem, without crossing over into other mounted partitions. (上述指令只會複製單一分割區,不會跨越掛載的其他分割區。) It will properly handle any special files, and it will completely preserve permissions.

If you have other filesystems mounted on other partitions, you do something somewhat similar. Say I had /boot mounted on a different filesystem, and I wanted to back it up onto /mnt/temp:

cd boot; find ./ -xdev -print0 | cpio -pa0V /mnt/temp

You simply would mount each new partition you want to copy onto, to /mnt/temp and repeat this above operation for each of those. Upon execution, you will see a series of dots fly across the screen. Each dot represents a file that is being copied. I often will go to another terminal and run watch df to watch the progress.

After all the filesystems have been copied, you probably won't actually have a bootloader. What I usually do, is go ahead and move the new drive into the place of the old one, and boot up on a Knoppix disc. Then I mount the new drive, say at /mnt/hda1, and then run grub-install –root-directory=/mnt/hda1 /dev/hda to copy my new bootloader to the new hard drive. You could probably do something similar just from your old install.

Backup and Restore

You can also use this method for a backup and restore. For example, say you want to back up your current / on machine1 onto a different Linux machine, machine2, temporarily at /home/foo/drivebackup/ (to resize your current partition or perhaps to format to a new filesytem). First take the drive from machine2 to machine1, mount machine2's drive at /mnt/temp, and run:

find / -xdev -print0 | cpio -pa0V /mnt/temp/home/foo/drivebackup

Once you do whatever it is you need to do on machine1, take the hard drive to machine2 (as repartitioning or formatting would have blanked it obviously) mount that hard drive on /mnt/temp and then run:

cd /home/foo/drivebackup; find ./ -xdev -print0 | cpio -pa0V /mnt/temp/

This will copy all the data back onto your old drive.

使用 dd

Cloning a partition \ From physical disk /dev/sda, partition 1, to physical disk /dev/sdb, partition 1:

# dd if=/dev/sda1 of=/dev/sdb1 bs=64K conv=noerror,sync status=progress

使用 tar

參考 https://help.ubuntu.com/community/BackupYourSystem/TAR

cd / 
tar -cvpzf /mnt/hdd/root_backup.tar.gz --one-file-system \
    --exclude=/proc \
    --exclude=/tmp \
    --exclude=/mnt \
    --exclude=/dev \
    --exclude=/sys \
    --exclude=/run \
    --exclude=/media \
    --exclude=/var/log \
    --exclude=/var/cache \
    --exclude=/var/run   / 

To understand what is going on, we will dissect each part of the command.

  1. tar - is the command that creates the archive. It is modified by each letter immediately following, each is explained bellow.
    • c - create a new backup archive.
    • v - verbose mode, tar will print what it's doing to the screen.
    • p - preserves the permissions of the files put in the archive for restoration later.
    • z - compress the backup file with 'gzip' to make it smaller.
    • f <filename> - specifies where to store the backup, backup.tar.gz is the filename used in this example. It will be stored in the current working directory, the one you set when you used the cd command.
  2. –exclude=/example/path - The options following this model instruct tar what directories NOT to backup. We don't want to backup everything since some directories aren't very useful to include. The first exclusion rule directs tar not to back itself up, this is important to avoid errors during the operation.
  3. –one-file-system - Do not include files on a different filesystem. If you want other filesystems, such as a /home partition, or external media mounted in /media backed up, you either need to back them up separately, or omit this flag. If you do omit this flag, you will need to add several more –exclude= arguments to avoid filesystems you do not want. These would be /proc, /sys, /mnt, /media, /run and /dev directories in root. /proc and /sys are virtual filesystems that provide windows into variables of the running kernel, so you do not want to try and backup or restore them. /dev is a tmpfs whose contents are created and deleted dynamically by udev, so you also do not want to backup or restore it. Likewise, /run is a tmpfs that holds variables about the running system that do not need backed up.
  4. It is important to note that these exclusions are recursive. This means that all folders located within the one excluded will be ignored as well. In the example, excluding the /media folder excludes all mounted drives and media there.
    • If there are certain partitions you wish to backup located in /media, simply remove the exclusion and write a new one excluding the partitions you don't want backed up stored within it. For example: –exclude=/media/unwanted_partition
  5. / - After all options is the directory to backup. Since we want to backup everything on the system we use / for the root directory. Like exclusions, this recursively includes every folder below root not listed in the exclusions or other options.

Linux file system 選擇

複製的目標檔案系統不需和原來的一樣,可以挑想要的檔案系統

關於 file system 的效能,可以參考 http://www.phoronix.com/scan.php?page=article&item=linux-40-hdd&num=1

  • XFS: 有不穩定的傾向,不要試了
  • ReiserFS: 一直留在 Gentoo 安裝指引的名單上,讓我一直想試。唉,試過之後,發現不支援 fstrim,在 SSD 上,會有問題。只好重新 format 成 ext4,最保險了。
  • Ext4 is the recommended all-purpose all-platform filesystem. 選它,大概就是一個四平八穩的方案。

目前硬碟的使用情形如下

/,根目錄,目前用了大概  40GB,就設 80GB 吧
/boot,使用 30MB,設 120 MB 吧。
swap,先前沒設? 好像也沒問題,就不需要了嗎

mkfs.ext4 /dev/sdb1
mkfs.ext4 /dev/sdb2

要 copy 整個 partition,有幾種作法,包括 dd,rsync,cpio。在下面的網址找到 cpio 的作法,就用它吧。
http://greenfly.net/tips/filesystem_migration.html

因為在 root 目錄下,有一些是目錄是 mount 其他 partition,所以,就再重新 mount 在另外的目錄下,再進行 copy。

mount /dev/sda2 /mnt/org
mount /dev/sdb2 /mnt/temp
cd  /mnt/org
find ./ -xdev -print0 | cpio -pa0V /mnt/temp
umount /mnt/org
umount /mnt/temp

mount /dev/sda1 /boot
mount /dev/sdb1 /mnt/temp
cd /boot
find ./ -xdev -print0 | cpio -pa0V /mnt/temp
umount /mnt/temp

執行 GRUB,我還是用舊的 0.97 版,其缺點是不支援 gpt,也就是硬碟超過 2TB 會有問題。但目前還可用,就將就著吧。
硬碟代碼,sda = hd0,sdb = hd1,sdc = hd2。
進入 grub 後,執行安裝指令
grub> root (hd1,0)
grub> setup (hd1)
grub> quit

接下來,就週整硬碟順序,就 OK 了。

假若是進 chroot 來 cpio 系統,然後要用 chroot 進新的磁碟,/mnt/gentoo/proc 可以直接 umount,但使用 mount --rbind 來掛載的 sys 和 dev,要用下面的方式 umount。
# grep /mnt/gentoo/sys /proc/mounts | cut -f2 -d" " | sort -r | xargs umount -n
# grep /mnt/gentoo/dev /proc/mounts | cut -f2 -d" " | sort -r | xargs umount -n


複製 CentOS 系統

以下針對 CentOS 7.9,而且沒有使用 lvm。參考 [Basic System Recovery]。

分割區的規劃如下。

# df -h
檔案系統        容量  已用  可用 已用% 掛載點
/dev/sdb3        74G  5.7G   64G    9% /
/dev/sdb1       926M  226M  637M   27% /boot
/dev/sdb5       375G  5.5G  351G    2% /home
掛載 及 chroot 指令。
# mkdir /mnt/sysimage
# mount /dev/sdb3 /mnt/sysimage
# mount /dev/sdb1 /mnt/sysimage/boot
# chroot /mnt/sysimage
掛但這樣是不夠的,在執行 grubs-install 指令時會出錯。
# grub2-install --target=i386-pc /dev/sdb
Installing for i386-pc platform.
grub2-install:錯誤: cannot find a device for /boot/grub2 (is /dev mounted?).

參考 Gentoo 的安裝指引。

# mount --types proc /proc /mnt/sysimage/proc
# mount --rbind /sys /mnt/sysimage/sys
# mount --make-rslave /mnt/sysimage/sys
# mount --rbind /dev /mnt/sysimage/dev
# mount --make-rslave /mnt/sysimage/dev
#
# chroot /mnt/sysimage /bin/bash
# source /etc/profile
# export PS1="(chroot) ${PS1}"

再次執行安裝 grub 指令,就不會出錯了。

# grub2-install --target=i386-pc /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.

接下來要設定開機選單。

# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-1160.25.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1160.25.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-1160.21.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1160.21.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-1160.6.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1160.6.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-ed726d3da8c142da88e77d06804e9ef0
Found initrd image: /boot/initramfs-0-rescue-ed726d3da8c142da88e77d06804e9ef0.img
done

列出產生的開機選單。

# awk -F\' '$1=="menuentry " {print i++ " : " $2}' /boot/grub2/grub.cfg
0 : CentOS Linux (3.10.0-1160.25.1.el7.x86_64) 7 (Core)
1 : CentOS Linux (3.10.0-1160.21.1.el7.x86_64) 7 (Core)
2 : CentOS Linux (3.10.0-1160.6.1.el7.x86_64) 7 (Core)
3 : CentOS Linux (0-rescue-ed726d3da8c142da88e77d06804e9ef0) 7 (Core)

檢查前次開機項次。

# grub2-editenv list
saved_entry=CentOS Linux (3.10.0-1160.25.1.el7.x86_64) 7 (Core)

最後,看新硬碟是否可以成功開機,若不成功,再重複檢查上述步驟。


2015年11月7日 星期六

調整 mp4 檔案的音量 (normalize)

雖然,avidemux 也可以做 normalize,但不會用,後來找到用 ffmpeg 就可以了。

參考連結
How can I normalize audio using ffmpeg?   其中 [Manually normalizing audio with ffmpeg]
Normalize audio in an avi file

Manually normalizing audio with ffmpeg

In ffmpeg you can use the volume filter to change the volume of a track. Make sure you download a recent version of the program.
This guide is for peak normalization, meaning that it will make the loudest part in the file sit at 0 dB instead of something lower. There is also RMS-based normalization which tries to make the average loudness the same across multiple files. To do that, do not try to push the maximum volume to 0 dB, but the mean volume to the dB level of choice (e.g. -26 dB).

Find out the gain to apply

First you need to analyze the audio stream for the maximum volume to see if normalizing would even pay off:
ffmpeg -i video.avi -af "volumedetect" -vn -sn -dn -f null /dev/null
Replace /dev/null with NUL on Windows.
The -vn-sn, and -dn arguments instruct ffmpeg to ignore non-audio streams during this analysis. This drastically speeds up the analysis.
This will output something like the following:
[Parsed_volumedetect_0 @ 0x7f8ba1c121a0] mean_volume: -16.0 dB
[Parsed_volumedetect_0 @ 0x7f8ba1c121a0] max_volume: -5.0 dB
[Parsed_volumedetect_0 @ 0x7f8ba1c121a0] histogram_0db: 87861
As you can see, our maximum volume is -5.0 dB, so we can apply 5 dB gain. If you get a value of 0 dB, then you don't need to normalize the audio.

Apply the volume filter:

Now we apply the volume filter to an audio file. Note that applying the filter means we will have to re-encode the audio stream. What codec you want for audio depends on the original format, of course. Here are some examples:
  • Plain audio file: Just encode the file with whatever encoder you need:
    ffmpeg -i input.wav -af "volume=5dB" output.mp3
    
    Your options are very broad, of course.
  • AVI format: Usually there's MP3 audio with video that comes in an AVI container:
    ffmpeg -i video.avi -af "volume=5dB" -c:v copy -c:a libmp3lame -q:a 2 output.avi
    
    Here we chose quality level 2. Values range from 0–9 and lower means better. Check the MP3 VBR guide for more info on setting the quality. You can also set a fixed bitrate with -b:a 192k, for example.
  • MP4 format: With an MP4 container, you will typically find AAC audio. We can use ffmpeg's build-in AAC encoder.
    ffmpeg -i video.mp4 -af "volume=5dB" -c:v copy -c:a aac -b:a 192k output.mp4
    
    Here you can also use other AAC encoders. Some of them support VBR, too. See this answer and the AAC encoding guide for some tips.
In the above examples, the video stream will be copied over using -c:v copy. If there are subtitles in your input file, or multiple video streams, use the option -map 0 before the output filename.

步驟摘要整理

找出聲音的大小,峰值為 -7.5 dB
ffmpeg -i v1.mp4 -af "volumedetect" -f null /dev/null
[Parsed_volumedetect_0 @ 0x1589580] n_samples: 397301760
[Parsed_volumedetect_0 @ 0x1589580] mean_volume: -40.3 dB
[Parsed_volumedetect_0 @ 0x1589580] max_volume: -7.5 dB

 將聲音提升 7.5 dB
ffmpeg -i v1.mp4 -af "volume=7.5dB" -c:v copy -c:a aac -strict experimental -b:a 128k v1n.mp4

處理後,得到的值
[Parsed_volumedetect_0 @ 0x2159720] n_samples: 397303808
[Parsed_volumedetect_0 @ 0x2159720] mean_volume: -32.9 dB
[Parsed_volumedetect_0 @ 0x2159720] max_volume: -0.2 dB

分離 mp3,另用 audacity 處理
ffmpeg -i ch1.mp4 -q:a 0 -map a ch1.mp3

https://www.youtube.com/watch?v=NgedD2zKhdc
Audacity Tutorial: Amplification, Compression, Normalizing and Exporting Your Podcast

使用 compressor 將大小聲做調整,不致動態太大。
http://manual.audacityteam.org/o/man/compressor.html

2015年11月5日 星期四

PHP -- 程式員的萬用瑞士刀

一般,談到 PHP,就是用來開發網頁吧。是的,尤其,PHP 中內建了伺服器,還可以用來當臨時的網頁伺服器,直接可以用來傳送檔案。
在Windows的環境下,相對於大部分的語言,都要安裝後才能用,PHP 則是下載,解壓縮後,即可使用。熟悉它的設定後,隨時隨地都以使用。
尤其,在Windows的環境下,要寫出功能強大的 batch 程式,幾乎是不可能任務。PHP 的 command mode 可用來執行 batch 程式。因此,PHP 真的可以說是程式人員的萬用工具。

網誌存檔